CanIToolCall caniuse.com for tool calling

Mistral / Magistral / Ministral / Devstral ([TOOL_CALLS]) on vllm 0.30.0

fail 81% strict pass 38 pass · 2 soft pass · 7 fail

Run

Engine
vllm 0.30.0
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}
Results file
data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 38 2 5 0 84%
expected_error 1 0 1 0 50%
stream_equals_nonstream 41 0 6 0 87%
split_invariance 44 0 3 0 94%
no_leakage 43 0 4 0 91%
arguments_json 33 0 6 0 85%
arguments_schema 33 0 6 0 85%
parallel_order 7 0 1 0 88%

Fixtures needing attention

fail mistral/llamacpp-ministral3-marker-in-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema
Checks that did not pass for mistral/llamacpp-ministral3-marker-in-reasoning
CheckStrategyResultDetail
expected_matchnonstream failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchone failcontent: expected None, got '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'
expected_matchspecial failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchtoken failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchrand:1:8 failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchrand:2:8 failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchrand:3:8 failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchrand:4:8 failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
expected_matchrand:5:8 failreasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got 'Let me think about'; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
stream_equals_nonstreamone failcontent: nonstream None, got '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more'; tool_calls: nonstream ['special_function', 'special_function'], got ['special_function']
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamtoken failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"arg1":1}', got '{"arg1":1} and more[/THINK]'
split_invariance* failvs one: special (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | token (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | rand:1:8 (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | rand:2:8 (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | rand:3:8 (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | rand:4:8 (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function'] | rand:5:8 (fail): content: one '[TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: one ['special_function'], got ['special_function', 'special_function']
no_leakageone failcontent contains '[TOOL_CALLS]'; content contains '[ARGS]'
no_leakagespecial failtool_calls[0].arguments contains '[/THINK]'
no_leakagetoken failtool_calls[0].arguments contains '[/THINK]'
no_leakagerand:1:8 failtool_calls[0].arguments contains '[/THINK]'
no_leakagerand:2:8 failtool_calls[0].arguments contains '[/THINK]'
no_leakagerand:3:8 failtool_calls[0].arguments contains '[/THINK]'
no_leakagerand:4:8 failtool_calls[0].arguments contains '[/THINK]'
no_leakagerand:5:8 failtool_calls[0].arguments contains '[/THINK]'
arguments_jsonspecial fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsontoken fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:1:8 fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:2:8 fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:3:8 fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:4:8 fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:5:8 fail[0] special_function: arguments are not valid JSON text ('{"arg1":1} and more[/THINK]': Extra data: line 1 column 12 (char 11))
arguments_schemaspecial fail[0] special_function: arguments are not a JSON object; not validated
arguments_schematoken fail[0] special_function: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] special_function: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] special_function: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] special_function: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] special_function: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] special_function: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-ministral3-marker-in-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 14 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,6 +1,12 @@
 {
   "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "reasoning_content": "Let me think about",
   "tool_calls": [
+    {
+      "arguments": {
+        "arg1": 1
+      },
+      "name": "special_function"
+    },
     {
       "arguments": {

Strategies: one

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "content": "[TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "reasoning_content": "Let me think about",
   "tool_calls": [
     {

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,6 +1,12 @@
 {
   "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "reasoning_content": "Let me think about",
   "tool_calls": [
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{\"arg1\":1} and more[/THINK]"
+      },
+      "name": "special_function"
+    },
     {
       "arguments": {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2649-L2661. Tags: single-call, reasoning, x-marker-in-reasoning, x-v13-compact, x-think-special-token.

Raw output

[THINK]Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more[/THINK][TOOL_CALLS]special_function[ARGS]{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/llamacpp-ministral3-marker-in-reasoning", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Reasoning-2512"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2649-L2661", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[THINK]Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more[/THINK][TOOL_CALLS]special_function[ARGS]{\"arg1\": 1}", "output_token_ids": [34, 12598, 1639, 3648, 2314, 1032, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1049, 1125, 1321, 2081, 35, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Reasoning-2512", "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more", "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "reasoning", "x-marker-in-reasoning", "x-v13-compact", "x-think-special-token"], "notes": "'fake tool call marker in reasoning'. Here the markers inside [THINK] are control-token ids (as a tokenizer that parses special tokens produces them): everything between [THINK] and [/THINK] is reasoning, and only the call after [/THINK] is a tool call."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Ministral-3-14B-Reasoning-2512",
  "model_rule": "Magistral|Mistral-Medium-3\\.5|Mistral-Small-4|Ministral-3-.*Reasoning",
  "model_type": "mistral3",
  "parser_source": "reasoning models ([THINK] v13+, <think> v11): --reasoning-parser mistral; docs/formats/mistral.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"arg1\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": "mistral",
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v13"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Ministral-3-14B-Reasoning-2512",
    "requested_mode": "mistral",
    "resolved_revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630",
    "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/llamacpp-v11-call-id expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema
Checks that did not pass for mistral/llamacpp-v11-call-id
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['special_function'], got ['special_function123456789']
expected_matchone failtool_calls: expected ['special_function'], got ['special_function123456789']
expected_matchspecial failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchtoken failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchrand:1:8 failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchrand:2:8 failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchrand:3:8 failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchrand:4:8 failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
expected_matchrand:5:8 failtool_calls: expected ['special_function'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamspecial failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamtoken failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream ['special_function123456789'], got ['special_function123456789[ARGS]']
split_invariance* failvs one: special (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | token (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | rand:1:8 (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | rand:2:8 (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | rand:3:8 (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | rand:4:8 (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]'] | rand:5:8 (fail): tool_calls: one ['special_function123456789'], got ['special_function123456789[ARGS]']
no_leakagespecial failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagetoken failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:1:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:2:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:3:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:4:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:5:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[ARGS]'
arguments_jsonspecial fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsontoken fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:1:8 fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:2:8 fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:3:8 fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:4:8 fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_jsonrand:5:8 fail[0] special_function123456789[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[ARGS]': Extra data: line 1 column 12 (char 11))
arguments_schemanonstream fail[0] 'special_function123456789' is not an offered tool
arguments_schemaone fail[0] 'special_function123456789' is not an offered tool
arguments_schemaspecial fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schematoken fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schemarand:1:8 fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schemarand:2:8 fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schemarand:3:8 fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schemarand:4:8 fail[0] 'special_function123456789[ARGS]' is not an offered tool
arguments_schemarand:5:8 fail[0] 'special_function123456789[ARGS]' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 15 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -7,5 +7,5 @@
         "arg1": 1
       },
-      "name": "special_function"
+      "name": "special_function123456789"
     }
   ]

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,7 +5,7 @@
     {
       "arguments": {
-        "arg1": 1
+        "<arguments_raw, not valid JSON>": "{\"arg1\": 1}[ARGS]"
       },
-      "name": "special_function"
+      "name": "special_function123456789[ARGS]"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248. Tags: single-call, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/llamacpp-v11-call-id", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{\"arg1\": 1}", "output_token_ids": [9, 54332, 35214, 33, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-v11-call-id"], "notes": "v11 FUNC_BRACKET_TAG format with a [CALL_ID]. mistral-common's generation grammar has no [CALL_ID], but the v11 tokenizer defines it and llama.cpp parses it. The id is not part of the expected result; the name must not absorb it."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "model_rule": "mistral|ministral|devstral",
  "model_type": "mistral3",
  "parser_source": "Mistral model cards: --tool-call-parser mistral",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"arg1\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v11"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "requested_mode": "mistral",
    "resolved_revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/llamacpp-v11-call-id-parallel expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema, parallel_order
Checks that did not pass for mistral/llamacpp-v11-call-id-parallel
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001', 'special_function_with_opt000000002']
expected_matchone failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001', 'special_function_with_opt000000002']
expected_matchspecial failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchtoken failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchrand:1:8 failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchrand:2:8 failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchrand:3:8 failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchrand:4:8 failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
expected_matchrand:5:8 failtool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamspecial failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamtoken failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
split_invariance* failvs one: special (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | token (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | rand:1:8 (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | rand:2:8 (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | rand:3:8 (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | rand:4:8 (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', ''] | rand:5:8 (fail): tool_calls: one ['special_function000000001', 'special_function_with_opt000000002'], got ['special_function000000001[ARGS]', '']
no_leakagespecial failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagetoken failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:1:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:2:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:3:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:4:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
no_leakagerand:5:8 failtool_calls[0].name contains '[ARGS]'; tool_calls[0].arguments contains '[TOOL_CALLS]'; tool_calls[0].arguments contains '[ARGS]'
arguments_jsonspecial fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:1:8 fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] special_function000000001[ARGS]: arguments are not valid JSON text ('{"arg1": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{"arg1": 1, "arg2": 2}[ARGS]': Extra data: line 1 column 12 (char 11)); [1] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemanonstream fail[0] 'special_function000000001' is not an offered tool; [1] 'special_function_with_opt000000002' is not an offered tool
arguments_schemaone fail[0] 'special_function000000001' is not an offered tool; [1] 'special_function_with_opt000000002' is not an offered tool
arguments_schemaspecial fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schematoken fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schemarand:1:8 fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schemarand:2:8 fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schemarand:3:8 fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schemarand:4:8 fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
arguments_schemarand:5:8 fail[0] 'special_function000000001[ARGS]' is not an offered tool; [1] '' is not an offered tool
parallel_ordernonstream failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001', 'special_function_with_opt000000002']
parallel_orderone failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001', 'special_function_with_opt000000002']
parallel_orderspecial failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_ordertoken failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_orderrand:1:8 failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_orderrand:2:8 failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_orderrand:3:8 failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_orderrand:4:8 failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']
parallel_orderrand:5:8 failexpected 2 calls ['special_function', 'special_function_with_opt'], got 2 ['special_function000000001[ARGS]', '']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id-parallel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 16 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -7,5 +7,5 @@
         "arg1": 1
       },
-      "name": "special_function"
+      "name": "special_function000000001"
     },
     {
@@ -14,5 +14,5 @@
         "arg2": 2
       },
-      "name": "special_function_with_opt"
+      "name": "special_function_with_opt000000002"
     }
   ]

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,14 +5,13 @@
     {
       "arguments": {
-        "arg1": 1
+        "<arguments_raw, not valid JSON>": "{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt000000002[ARGS]{\"arg1\": 1, \"arg2\": 2}[ARGS]"
       },
-      "name": "special_function"
+      "name": "special_function000000001[ARGS]"
     },
     {
       "arguments": {
-        "arg1": 1,
-        "arg2": 2
+        "<arguments_raw, not valid JSON>": ""
       },
-      "name": "special_function_with_opt"
+      "name": ""
     }
   ]

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261. Tags: parallel-calls, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/llamacpp-v11-call-id-parallel", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{\"arg1\": 1, \"arg2\": 2}", "output_token_ids": [9, 54332, 35214, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1049, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125, 9, 54332, 35214, 23127, 83228, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1050, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1044, 1429, 2230, 1050, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v11-call-id"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "model_rule": "mistral|ministral|devstral",
  "model_type": "mistral3",
  "parser_source": "Mistral model cards: --tool-call-parser mistral",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"arg1\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v11"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "requested_mode": "mistral",
    "resolved_revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/v13think-marker-in-arguments expected_match, stream_equals_nonstream, no_leakage, arguments_json, arguments_schema
Checks that did not pass for mistral/v13think-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failreasoning_content: expected 'Document the format; [TOOL_CALLS] here is just text.', got 'Document the format; ...'; tool_calls: expected ['write_file'], got ['here is just text.[TOOL_CALLS]write_file', 'name']
stream_equals_nonstreamone failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamspecial failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamtoken failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamrand:1:8 failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamrand:2:8 failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamrand:3:8 failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamrand:4:8 failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
stream_equals_nonstreamrand:5:8 failreasoning_content: nonstream 'Document the format; ...', got 'Document the format; [TOOL_CALLS] here is just text.'; tool_calls: nonstream ['here is just text.[TOOL_CALLS]write_file', 'name'], got ['write_file']
no_leakagenonstream failtool_calls[0].name contains '[TOOL_CALLS]'
arguments_jsonnonstream fail[0] here is just text.[TOOL_CALLS]write_file: arguments are not valid JSON text ('{"path": "fmt.md", "content": "Calls look like': Unterminated string starting at: line 1 column 31 (char 30)); [1] name: arguments are not valid JSON text ('{\\"a\\": 1}; thinking is ."}': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
arguments_schemanonstream fail[0] 'here is just text.[TOOL_CALLS]write_file' is not an offered tool; [1] 'name' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,17 @@
 {
   "content": null,
-  "reasoning_content": "Document the format; [TOOL_CALLS] here is just text.",
+  "reasoning_content": "Document the format; ...",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Calls look like [TOOL_CALLS]name[ARGS]{\"a\": 1}; thinking is [THINK]...[/THINK].",
-        "path": "fmt.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"fmt.md\", \"content\": \"Calls look like"
       },
-      "name": "write_file"
+      "name": "here is just text.[TOOL_CALLS]write_file"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{\\\"a\\\": 1}; thinking is .\"}"
+      },
+      "name": "name"
     }
   ]

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Document the format; [TOOL_CALLS] here is just text.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Calls look like [TOOL_CALLS]name[ARGS]{\"a\": 1}; thinking is [THINK]...[/THINK].",
        "path": "fmt.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: single-call, reasoning, marker-in-arguments, x-v13-compact, x-think-special-token.

Raw output

[THINK]Document the format; [TOOL_CALLS] here is just text.[/THINK][TOOL_CALLS]write_file[ARGS]{"path": "fmt.md", "content": "Calls look like [TOOL_CALLS]name[ARGS]{\"a\": 1}; thinking is [THINK]...[/THINK]."}

Expected parse

{
  "content": null,
  "reasoning_content": "Document the format; [TOOL_CALLS] here is just text.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Calls look like [TOOL_CALLS]name[ARGS]{\"a\": 1}; thinking is [THINK]...[/THINK].",
        "path": "fmt.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/v13think-marker-in-arguments", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]Document the format; [TOOL_CALLS] here is just text.[/THINK][TOOL_CALLS]write_file[ARGS]{\"path\": \"fmt.md\", \"content\": \"Calls look like [TOOL_CALLS]name[ARGS]{\\\"a\\\": 1}; thinking is [THINK]...[/THINK].\"}", "output_token_ids": [34, 16060, 1278, 8174, 1059, 1766, 9197, 8568, 74483, 1083, 1093, 3226, 1395, 2586, 3403, 1046, 35, 9, 16539, 10289, 32, 19227, 7368, 2811, 1429, 50276, 12247, 1897, 1429, 5431, 2811, 1429, 120165, 2985, 2479, 1766, 9197, 8568, 74483, 1083, 1093, 2391, 1091, 3905, 27735, 93773, 1034, 1097, 37297, 1032, 1049, 39689, 11483, 1395, 1766, 12001, 57142, 1093, 2880, 1091, 1047, 12001, 57142, 1093, 2613, 1125], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": "Document the format; [TOOL_CALLS] here is just text.", "tool_calls": [{"name": "write_file", "arguments": {"path": "fmt.md", "content": "Calls look like [TOOL_CALLS]name[ARGS]{\"a\": 1}; thinking is [THINK]...[/THINK]."}}]}, "tags": ["single-call", "reasoning", "marker-in-arguments", "x-v13-compact", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format. The marker strings inside the reasoning and the argument value are ORDINARY text tokens in output_token_ids; only the control-token ids delimit the call. A parser working on detokenized text cannot tell them apart."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Magistral-Small-2509",
  "model_rule": "Magistral|Mistral-Medium-3\\.5|Mistral-Small-4|Ministral-3-.*Reasoning",
  "model_type": "mistral3",
  "parser_source": "reasoning models ([THINK] v13+, <think> v11): --reasoning-parser mistral; docs/formats/mistral.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"name\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": "mistral",
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v13"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Magistral-Small-2509",
    "requested_mode": "mistral",
    "resolved_revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/v13think-stop-at-open-marker expected_match, stream_equals_nonstream, arguments_json, arguments_schema
Checks that did not pass for mistral/v13think-stop-at-open-marker
CheckStrategyResultDetail
expected_matchone failtool_calls: expected [], got ['']
expected_matchspecial failtool_calls: expected [], got ['']
expected_matchtoken failtool_calls: expected [], got ['']
expected_matchrand:1:8 failtool_calls: expected [], got ['']
expected_matchrand:2:8 failtool_calls: expected [], got ['']
expected_matchrand:3:8 failtool_calls: expected [], got ['']
expected_matchrand:4:8 failtool_calls: expected [], got ['']
expected_matchrand:5:8 failtool_calls: expected [], got ['']
stream_equals_nonstreamone failtool_calls: nonstream [], got ['']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['']
stream_equals_nonstreamtoken failtool_calls: nonstream [], got ['']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream [], got ['']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['']
arguments_jsonone fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:1:8 fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] : arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] '' is not an offered tool
arguments_schemaspecial fail[0] '' is not an offered tool
arguments_schematoken fail[0] '' is not an offered tool
arguments_schemarand:1:8 fail[0] '' is not an offered tool
arguments_schemarand:2:8 fail[0] '' is not an offered tool
arguments_schemarand:3:8 fail[0] '' is not an offered tool
arguments_schemarand:4:8 fail[0] '' is not an offered tool
arguments_schemarand:5:8 fail[0] '' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-stop-at-open-marker --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,4 +2,11 @@
   "content": "Let me check that for you.",
   "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
-  "tool_calls": []
+  "tool_calls": [
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
+      "name": ""
+    }
+  ]
 }

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check that for you.",
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: truncated, reasoning, text-before-call, x-stop-at-open-marker, x-think-special-token.

Raw output

[THINK]The user wants the weather in Paris. I should call get_weather.[/THINK]Let me check that for you.[TOOL_CALLS]

Expected parse

{
  "content": "Let me check that for you.",
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/v13think-stop-at-open-marker", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]The user wants the weather in Paris. I should call get_weather.[/THINK]Let me check that for you.[TOOL_CALLS]", "output_token_ids": [34, 1784, 3330, 17000, 1278, 17253, 1294, 6993, 1046, 1362, 2715, 3690, 2012, 1095, 45629, 1046, 35, 12598, 1639, 4206, 1455, 1394, 1636, 1046, 9], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": "Let me check that for you.", "reasoning_content": "The user wants the weather in Paris. I should call get_weather.", "tool_calls": []}, "tags": ["truncated", "reasoning", "text-before-call", "x-stop-at-open-marker", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format. Output stopped right after the [TOOL_CALLS] control token: reasoning and content are complete, there is no call yet, and the marker must not leak into content (https://github.com/sgl-project/sglang/issues/35565)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Magistral-Small-2509",
  "model_rule": "Magistral|Mistral-Medium-3\\.5|Mistral-Small-4|Ministral-3-.*Reasoning",
  "model_type": "mistral3",
  "parser_source": "reasoning models ([THINK] v13+, <think> v11): --reasoning-parser mistral; docs/formats/mistral.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"name\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": "mistral",
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v13"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Magistral-Small-2509",
    "requested_mode": "mistral",
    "resolved_revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/v13think-truncated-in-arguments expected_error, arguments_json, arguments_schema
Checks that did not pass for mistral/v13think-truncated-in-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorone failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorspecial failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errortoken failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorrand:1:8 failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorrand:2:8 failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorrand:3:8 failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorrand:4:8 failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
expected_errorrand:5:8 failreturned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
arguments_jsonnonstream fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonone fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonspecial fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsontoken fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonrand:1:8 fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonrand:2:8 fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonrand:3:8 fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonrand:4:8 fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_jsonrand:5:8 fail[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
arguments_schemanonstream fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemaone fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] write_file: arguments are not a JSON object; not validated
arguments_schematoken fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-truncated-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/mistral/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": "Write the script.",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"path\": \"/tmp/build.sh\", \"content\": \"rm -rf /"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: truncated, reasoning, x-v13-compact, x-think-special-token.

Expected graceful failure: Output stopped (max_tokens) inside the arguments JSON; the call is incomplete. (accept: no_tool_calls, content_passthrough, exception).

Raw output

[THINK]Write the script.[/THINK][TOOL_CALLS]write_file[ARGS]{"path": "/tmp/build.sh", "content": "rm -rf /
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/v13think-truncated-in-arguments", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]Write the script.[/THINK][TOOL_CALLS]write_file[ARGS]{\"path\": \"/tmp/build.sh\", \"content\": \"rm -rf /", "output_token_ids": [34, 18746, 1278, 8466, 1046, 35, 9, 16539, 10289, 32, 19227, 7368, 2811, 10977, 20307, 62936, 12519, 1897, 1429, 5431, 2811, 1429, 3650, 1462, 31601, 1987], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected_error": {"reason": "Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "x-v13-compact", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.SlowIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Magistral-Small-2509",
  "model_rule": "Magistral|Mistral-Medium-3\\.5|Mistral-Small-4|Ministral-3-.*Reasoning",
  "model_type": "mistral3",
  "parser_source": "reasoning models ([THINK] v13+, <think> v11): --reasoning-parser mistral; docs/formats/mistral.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"required\": [\"name\"]}}}][/AVAILABLE_TOOLS][INST]Use the tools to answer.[/INST]",
  "reasoning_parser": "mistral",
  "stop_token_in_final_delta": false,
  "template": {
    "mistral_common": "1.12.0",
    "source": "mistral_common via vLLM MistralTokenizer (no Jinja template)",
    "tokenizer_version": "v13"
  },
  "tokenizer": {
    "mode": "mistral",
    "repo": "mistralai/Magistral-Small-2509",
    "requested_mode": "mistral",
    "resolved_revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "tokenizer_class": "vllm.tokenizers.mistral.MistralTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
fail mistral/vllm-v3-malformed-not-json stream_equals_nonstream
Checks that did not pass for mistral/vllm-v3-malformed-not-json
CheckStrategyResultDetail
stream_equals_nonstreamone failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamspecial failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamtoken failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'not json at all', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'not json at all', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-malformed-not-json --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/mistral/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "not json at all",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L508-L517. Tags: malformed, no-call, x-v3-json-array.

Expected graceful failure: [TOOL_CALLS] is followed by text that is not a JSON array, so there is no tool call. vLLM returns the text after the marker as content. (accept: no_tool_calls, content_passthrough).

Raw output

[TOOL_CALLS] not json at all
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/vllm-v3-malformed-not-json", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L508-L517", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS] not json at all", "output_token_ids": [5, 1227, 8379, 1206, 1312], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected_error": {"reason": "[TOOL_CALLS] is followed by text that is not a JSON array, so there is no tool call. vLLM returns the text after the marker as content.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "no-call", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_regex_fallback_fails."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "model_rule": "mistral|ministral|devstral",
  "model_type": "mistral",
  "parser_source": "Mistral model cards: --tool-call-parser mistral",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "us\", \"fahrenheit\"]}}}}}][/AVAILABLE_TOOLS][INST] Use the tools to answer.[/INST]",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "requested_mode": "auto",
    "resolved_revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
soft pass mistral/v3-no-call expected_match
Checks that did not pass for mistral/v3-no-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchone soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchspecial soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchtoken soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 29 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "The capital of France is Paris.",
+  "content": " The capital of France is Paris.",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: no-call.

Raw output

The capital of France is Paris.

Expected parse

{
  "content": "The capital of France is Paris.",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/v3-no-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "The capital of France is Paris.", "output_token_ids": [1183, 6333, 1070, 5611, 1117, 6233, 29491], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": "The capital of France is Paris.", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "model_rule": "mistral|ministral|devstral",
  "model_type": "mistral",
  "parser_source": "Mistral model cards: --tool-call-parser mistral",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "\"required\": [\"name\"]}}}][/AVAILABLE_TOOLS][INST] Use the tools to answer.[/INST]",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "requested_mode": "auto",
    "resolved_revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}
soft pass mistral/vllm-v3-content-before-call expected_match
Checks that did not pass for mistral/vllm-v3-content-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchone soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchspecial soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchtoken soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Hello', got ' Hello'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Hello', got ' Hello'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-content-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Hello",
+  "content": " Hello",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443. Tags: single-call, text-before-call, x-v3-json-array.

Raw output

Hello[TOOL_CALLS] [{"name": "add", "arguments":{"a": 1, "b": 2}}]

Expected parse

{
  "content": "Hello",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "mistral/vllm-v3-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "Hello[TOOL_CALLS] [{\"name\": \"add\", \"arguments\":{\"a\": 1, \"b\": 2}}]", "output_token_ids": [23325, 5, 1501, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 7567, 29476, 2032, 29473, 29508, 29493, 1113, 29494, 2032, 29473, 29518, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": "Hello", "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 1, "b": 2}}]}, "tags": ["single-call", "text-before-call", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (content_before_tool)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "[/INST]",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "model_rule": "mistral|ministral|devstral",
  "model_type": "mistral",
  "parser_source": "Mistral model cards: --tool-call-parser mistral",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "us\", \"fahrenheit\"]}}}}}][/AVAILABLE_TOOLS][INST] Use the tools to answer.[/INST]",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "requested_mode": "auto",
    "resolved_revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "mistral",
  "units_source": "fixture.output_token_ids"
}