Mistral / Magistral / Ministral / Devstral ([TOOL_CALLS]) on sglang 0.5.20

fail 49% strict pass 23 pass · 1 soft pass · 23 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	21	1	23	47%
`expected_error`	2	0	0	100%
`stream_equals_nonstream`	26	0	21	55%
`split_invariance`	27	0	20	57%
`no_leakage`	42	0	5	89%
`arguments_json`	27	0	7	79%
`arguments_schema`	27	0	7	79%
`parallel_order`	0	0	8	0%

Fixtures needing attention

fail mistral/llamacpp-devstral-content-before-call expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for mistral/llamacpp-devstral-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['special_function'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['special_function'], got []
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['special_function'] \| token (fail): tool_calls: one [], got ['special_function'] \| rand:1:8 (fail): tool_calls: one [], got ['special_function'] \| rand:2:8 (fail): tool_calls: one [], got ['special_function'] \| rand:3:8 (fail): tool_calls: one [], got ['special_function'] \| rand:4:8 (fail): tool_calls: one [], got ['special_function'] \| rand:5:8 (fail): tool_calls: one [], got ['special_function']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-devstral-content-before-call --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": "Hello, world!\nWhat's up?",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Hello, world!\nWhat's up?",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6280-L6284. Tags: single-call, text-before-call, numeric-arguments, x-v11-args.

Raw output

Hello, world!
What's up?[TOOL_CALLS]special_function[ARGS]{"arg1": 1}

Expected parse

{
  "content": "Hello, world!\nWhat's up?",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-devstral-content-before-call", "family": "mistral", "models": ["mistralai/Devstral-Small-2507"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6280-L6284", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "Hello, world!\nWhat's up?[TOOL_CALLS]special_function[ARGS]{\"arg1\": 1}", "output_token_ids": [22177, 1044, 4304, 11330, 7493, 1681, 2015, 1063, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Devstral-Small-2507", "revision": "bd165ab26cebbcc2eea2c4ecbfc07f3ac42b3c39", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "Hello, world!\nWhat's up?", "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "text-before-call", "numeric-arguments", "x-v11-args"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Devstral-Small-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Devstral-Small-2507",
    "revision": "bd165ab26cebbcc2eea2c4ecbfc07f3ac42b3c39",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/llamacpp-ministral3-marker-in-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for mistral/llamacpp-ministral3-marker-in-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '[THINK]Let me think about'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`one`	fail	content: expected None, got '[THINK]Let me think about '; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	content: expected None, got '[THINKLet me think about and more[/THINK'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`token`	fail	content: expected None, got '[THINKLet me think about and more[/THINK'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`rand:1:8`	fail	content: expected None, got '[THINKLet me think about and more[/THINK]'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`rand:2:8`	fail	content: expected None, got '[THINKLet me think about and more[/THINK]'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`rand:3:8`	fail	content: expected None, got '[THINKLet me think about and more[/THINK]'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`rand:4:8`	fail	content: expected None, got '[THINKLet me think about and more[/THINK]'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`expected_match`	`rand:5:8`	fail	content: expected None, got '[THINKLet me think about and more[/THINK]'; reasoning_content: expected 'Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more', got None; tool_calls: expected ['special_function'], got ['special_function', 'special_function']
`stream_equals_nonstream`	`one`	fail	content (whitespace only): nonstream '[THINK]Let me think about', got '[THINK]Let me think about '; tool_calls: nonstream ['special_function', 'special_function'], got []
`stream_equals_nonstream`	`special`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK'
`stream_equals_nonstream`	`token`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK]'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK]'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK]'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK]'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '[THINK]Let me think about', got '[THINKLet me think about and more[/THINK]'
`split_invariance`	`*`	fail	vs one: special (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK'; tool_calls: one [], got ['special_function', 'special_function'] \| token (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK'; tool_calls: one [], got ['special_function', 'special_function'] \| rand:1:8 (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK]'; tool_calls: one [], got ['special_function', 'special_function'] \| rand:2:8 (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK]'; tool_calls: one [], got ['special_function', 'special_function'] \| rand:3:8 (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK]'; tool_calls: one [], got ['special_function', 'special_function'] \| rand:4:8 (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK]'; tool_calls: one [], got ['special_function', 'special_function'] \| rand:5:8 (fail): content: one '[THINK]Let me think about ', got '[THINKLet me think about and more[/THINK]'; tool_calls: one [], got ['special_function', 'special_function']
`no_leakage`	`nonstream`	fail	content contains '[THINK]'
`no_leakage`	`one`	fail	content contains '[THINK]'
`no_leakage`	`rand:1:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:2:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:3:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:4:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:5:8`	fail	content contains '[/THINK]'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-ministral3-marker-in-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,6 +1,12 @@
 {
-  "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "content": "[THINK]Let me think about",
+  "reasoning_content": null,
   "tool_calls": [
+    {
+      "arguments": {
+        "arg1": 1
+      },
+      "name": "special_function"
+    },
     {
       "arguments": {

Strategies: one

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": "[THINK]Let me think about ",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: special, token

--- expected
+++ observed
@@ -1,6 +1,12 @@
 {
-  "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "content": "[THINKLet me think about  and more[/THINK",
+  "reasoning_content": null,
   "tool_calls": [
+    {
+      "arguments": {
+        "arg1": 1
+      },
+      "name": "special_function"
+    },
     {
       "arguments": {

Strategies: rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,6 +1,12 @@
 {
-  "content": null,
-  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
+  "content": "[THINKLet me think about  and more[/THINK]",
+  "reasoning_content": null,
   "tool_calls": [
+    {
+      "arguments": {
+        "arg1": 1
+      },
+      "name": "special_function"
+    },
     {
       "arguments": {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2649-L2661. Tags: single-call, reasoning, x-marker-in-reasoning, x-v13-compact, x-think-special-token.

Raw output

[THINK]Let me think about [TOOL_CALLS]special_function[ARGS]{"arg1":1} and more[/THINK][TOOL_CALLS]special_function[ARGS]{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more",
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-ministral3-marker-in-reasoning", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Reasoning-2512"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2649-L2661", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[THINK]Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more[/THINK][TOOL_CALLS]special_function[ARGS]{\"arg1\": 1}", "output_token_ids": [34, 12598, 1639, 3648, 2314, 1032, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1049, 1125, 1321, 2081, 35, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Reasoning-2512", "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": "Let me think about [TOOL_CALLS]special_function[ARGS]{\"arg1\":1} and more", "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "reasoning", "x-marker-in-reasoning", "x-v13-compact", "x-think-special-token"], "notes": "'fake tool call marker in reasoning'. Here the markers inside [THINK] are control-token ids (as a tokenizer that parses special tokens produces them): everything between [THINK] and [/THINK] is reasoning, and only the call after [/THINK] is a tool call."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Ministral-3-14B-Reasoning-2512",
  "no_stop_trim": false,
  "notes": [
    "thinking=True not applied: Anthropic thinking is not supported for models without a reasoning parser"
  ],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Ministral-3-14B-Reasoning-2512",
    "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/llamacpp-ministral3-parallel expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for mistral/llamacpp-ministral3-parallel
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['special_function', 'special_function_with_opt'], got ['special_function']
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| token (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| rand:1:8 (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| rand:2:8 (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| rand:3:8 (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| rand:4:8 (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt'] \| rand:5:8 (fail): tool_calls: one ['special_function'], got ['special_function', 'special_function_with_opt']
`parallel_order`	`one`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 1 ['special_function']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-ministral3-parallel --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 13 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -8,11 +8,4 @@
       },
       "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
     }
   ]

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2624-L2636. Tags: parallel-calls, numeric-arguments, x-v13-compact.

Raw output

[TOOL_CALLS]special_function[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[ARGS]{"arg1": 1, "arg2": 2}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-ministral3-parallel", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Reasoning-2512"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2624-L2636", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "[TOOL_CALLS]special_function[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[ARGS]{\"arg1\": 1, \"arg2\": 2}", "output_token_ids": [9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125, 9, 54332, 35214, 23127, 83228, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1044, 1429, 2230, 1050, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Reasoning-2512", "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v13-compact"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Ministral-3-14B-Reasoning-2512",
  "no_stop_trim": false,
  "notes": [
    "thinking=True not applied: Anthropic thinking is not supported for models without a reasoning parser"
  ],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Ministral-3-14B-Reasoning-2512",
    "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail mistral/llamacpp-ministral3-reasoning-call expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for mistral/llamacpp-ministral3-reasoning-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got "[THINK]I'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`one`	fail	content: expected None, got "[THINK]I'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None; tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`token`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`rand:1:8`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`rand:2:8`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`rand:3:8`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`rand:4:8`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None
`expected_match`	`rand:5:8`	fail	content: expected None, got "[THINKI'm\nthinking[/THINK]"; reasoning_content: expected "I'm\nthinking", got None
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['special_function'], got []
`stream_equals_nonstream`	`special`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"
`stream_equals_nonstream`	`token`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"
`split_invariance`	`*`	fail	vs one: special (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"; tool_calls: one [], got ['special_function'] \| token (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"; tool_calls: one [], got ['special_function'] \| rand:1:8 (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"; tool_calls: one [], got ['special_function'] \| rand:2:8 (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"; tool_calls: one [], got ['special_function'] \| rand:3:8 (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK"; tool_calls: one [], got ['special_function'] \| rand:4:8 (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"; tool_calls: one [], got ['special_function'] \| rand:5:8 (fail): content: one "[THINK]I'm\nthinking[/THINK]", got "[THINKI'm\nthinking[/THINK]"; tool_calls: one [], got ['special_function']
`no_leakage`	`nonstream`	fail	content contains '[THINK]'; content contains '[/THINK]'
`no_leakage`	`one`	fail	content contains '[THINK]'; content contains '[/THINK]'
`no_leakage`	`rand:1:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:2:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:4:8`	fail	content contains '[/THINK]'
`no_leakage`	`rand:5:8`	fail	content contains '[/THINK]'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-ministral3-reasoning-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I'm\nthinking",
+  "content": "[THINK]I'm\nthinking[/THINK]",
+  "reasoning_content": null,
   "tool_calls": [
     {

Strategies: one

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I'm\nthinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": "[THINK]I'm\nthinking[/THINK]",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: special, token, rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I'm\nthinking",
+  "content": "[THINKI'm\nthinking[/THINK",
+  "reasoning_content": null,
   "tool_calls": [
     {

Strategies: rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I'm\nthinking",
+  "content": "[THINKI'm\nthinking[/THINK]",
+  "reasoning_content": null,
   "tool_calls": [
     {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2615-L2622. Tags: single-call, reasoning, numeric-arguments, x-v13-compact, x-think-special-token.

Raw output

[THINK]I'm
thinking[/THINK][TOOL_CALLS]special_function[ARGS]{"arg1":1}

Expected parse

{
  "content": null,
  "reasoning_content": "I'm\nthinking",
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-ministral3-reasoning-call", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Reasoning-2512"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2615-L2622", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[THINK]I'm\nthinking[/THINK][TOOL_CALLS]special_function[ARGS]{\"arg1\":1}", "output_token_ids": [34, 1073, 4525, 1010, 1411, 34053, 35, 9, 54332, 35214, 32, 19227, 2230, 1049, 2811, 1049, 1125], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Reasoning-2512", "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": "I'm\nthinking", "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "reasoning", "numeric-arguments", "x-v13-compact", "x-think-special-token"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Ministral-3-14B-Reasoning-2512",
  "no_stop_trim": false,
  "notes": [
    "thinking=True not applied: Anthropic thinking is not supported for models without a reasoning parser"
  ],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Ministral-3-14B-Reasoning-2512",
    "revision": "51f9210f3cd20f3452a80d5819d15dc61cc50630",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/llamacpp-v11-call-id expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for mistral/llamacpp-v11-call-id
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`one`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`token`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'; tool_calls: expected ['special_function'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}'
`no_leakage`	`one`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`special`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`token`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:1:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:2:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:3:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:4:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:5:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 15 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{\"arg1\": 1}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248. Tags: single-call, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-v11-call-id", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{\"arg1\": 1}", "output_token_ids": [9, 54332, 35214, 33, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-v11-call-id"], "notes": "v11 FUNC_BRACKET_TAG format with a [CALL_ID]. mistral-common's generation grammar has no [CALL_ID], but the v11 tokenizer defines it and llama.cpp parses it. The id is not part of the expected result; the name must not absorb it."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/llamacpp-v11-call-id-parallel expected_match, stream_equals_nonstream, split_invariance, no_leakage, parallel_order

Checks that did not pass for mistral/llamacpp-v11-call-id-parallel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`one`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`special`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`token`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'; tool_calls: expected ['special_function', 'special_function_with_opt'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'
`split_invariance`	`*`	fail	vs one: rand:1:8 (fail): content: one '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}', got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS{"arg1": 1, "arg2": 2}' \| rand:2:8 (fail): content: one '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}', got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}' \| rand:4:8 (fail): content: one '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}', got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}' \| rand:5:8 (fail): content: one '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}', got '[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{"arg1": 1, "arg2": 2}'
`no_leakage`	`one`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`special`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`token`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:1:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:2:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:3:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:4:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`no_leakage`	`rand:5:8`	fail	content contains '[TOOL_CALLS]'; content contains '[ARGS]'; content contains '[CALL_ID]'
`parallel_order`	`nonstream`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`one`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`special`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`rand:1:8`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`rand:2:8`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`rand:3:8`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`rand:4:8`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`rand:5:8`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id-parallel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -2,18 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:3:8

--- expected
+++ observed
@@ -1,19 +1,5 @@
 {
-  "content": null,
+  "content": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{\"arg1\": 1, \"arg2\": 2}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,19 +1,5 @@
 {
-  "content": null,
+  "content": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS{\"arg1\": 1, \"arg2\": 2}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,19 +1,5 @@
 {
-  "content": null,
+  "content": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID000000002[ARGS{\"arg1\": 1, \"arg2\": 2}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261. Tags: parallel-calls, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-v11-call-id-parallel", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{\"arg1\": 1, \"arg2\": 2}", "output_token_ids": [9, 54332, 35214, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1049, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125, 9, 54332, 35214, 23127, 83228, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1050, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1044, 1429, 2230, 1050, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v11-call-id"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail mistral/v11-content-before-call expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for mistral/v11-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_time'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_time'], got []
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['get_time'] \| token (fail): tool_calls: one [], got ['get_time'] \| rand:1:8 (fail): tool_calls: one [], got ['get_time'] \| rand:2:8 (fail): tool_calls: one [], got ['get_time'] \| rand:3:8 (fail): tool_calls: one [], got ['get_time'] \| rand:4:8 (fail): tool_calls: one [], got ['get_time'] \| rand:5:8 (fail): tool_calls: one [], got ['get_time']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v11-content-before-call --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 24 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": "Checking the clock.",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Checking the clock.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43. Tags: single-call, text-before-call, empty-arguments, x-v11-args.

Raw output

Checking the clock.[TOOL_CALLS]get_time[ARGS]{}

Expected parse

{
  "content": "Checking the clock.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v11-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "6e2501687ccd0e1f30f36319eaf2b46958b897811e246cd8eb5d385b9e3de7d1", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "Checking the clock.[TOOL_CALLS]get_time[ARGS]{}", "output_token_ids": [109758, 1278, 16475, 1046, 9, 1689, 11332, 32, 30620], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "Checking the clock.", "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "text-before-call", "empty-arguments", "x-v11-args"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-Small-3.2-24B-Instruct-2506 (tokenizer v11); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids; with ids, v11 renders [CALL_ID]<id> after the name. Whether the model generates [CALL_ID] is unverified (docs/formats/mistral.md covers both shapes)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v11-parallel-calls expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for mistral/v11-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v11-parallel-calls --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 23 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -8,13 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43. Tags: parallel-calls, unicode, nested-json, string-escapes, x-v11-args.

Raw output

[TOOL_CALLS]get_weather[ARGS]{"city": "Zürich"}[TOOL_CALLS]search[ARGS]{"query": "café \"best\"", "filters": {"max": 3}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v11-parallel-calls", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "6e2501687ccd0e1f30f36319eaf2b46958b897811e246cd8eb5d385b9e3de7d1", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Zürich\"}[TOOL_CALLS]search[ARGS]{\"query\": \"café \\\"best\\\"\", \"filters\": {\"max\": 3}}", "output_token_ids": [9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 1090, 2592, 1521, 46005, 9, 8928, 32, 19227, 5272, 2811, 1429, 3173, 1102, 1337, 25994, 22759, 17931, 1897, 1429, 77277, 2811, 16753, 5914, 2811, 1032, 1051, 2821], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-v11-args"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-Small-3.2-24B-Instruct-2506 (tokenizer v11); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids; with ids, v11 renders [CALL_ID]<id> after the name. Whether the model generates [CALL_ID] is unverified (docs/formats/mistral.md covers both shapes)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v13-content-before-call expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for mistral/v13-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['search'], got []
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['search'] \| token (fail): tool_calls: one [], got ['search'] \| rand:1:8 (fail): tool_calls: one [], got ['search'] \| rand:2:8 (fail): tool_calls: one [], got ['search'] \| rand:3:8 (fail): tool_calls: one [], got ['search'] \| rand:4:8 (fail): tool_calls: one [], got ['search'] \| rand:5:8 (fail): tool_calls: one [], got ['search']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13-content-before-call --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 13 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": "I'll search for that.",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vLLM release notes"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "I'll search for that.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "vLLM release notes"
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512/tree/29439f81c2be264d8d393273f99e7db9c0961120. Tags: single-call, text-before-call, x-v13-compact.

Raw output

I'll search for that.[TOOL_CALLS]search[ARGS]{"query": "vLLM release notes"}

Expected parse

{
  "content": "I'll search for that.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "vLLM release notes"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v13-content-before-call", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Instruct-2512", "mistralai/Devstral-Small-2-24B-Instruct-2512", "mistralai/Magistral-Small-2509"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512/tree/29439f81c2be264d8d393273f99e7db9c0961120", "revision": "29439f81c2be264d8d393273f99e7db9c0961120", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "600bb27946565481ecf51ba8aee252e49b9a68507866080ac9c30185bb312843", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "I'll search for that.[TOOL_CALLS]search[ARGS]{\"query\": \"vLLM release notes\"}", "output_token_ids": [1073, 7534, 6123, 1394, 1455, 1046, 9, 8928, 32, 19227, 5272, 2811, 1429, 1118, 23947, 1077, 8021, 15031, 46005], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Instruct-2512", "revision": "29439f81c2be264d8d393273f99e7db9c0961120", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "I'll search for that.", "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "vLLM release notes"}}]}, "tags": ["single-call", "text-before-call", "x-v13-compact"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Ministral-3-14B-Instruct-2512 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Ministral-3-14B-Instruct-2512",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Ministral-3-14B-Instruct-2512",
    "revision": "29439f81c2be264d8d393273f99e7db9c0961120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v13-parallel-calls expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for mistral/v13-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_time'], got ['get_weather']
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_time'], got ['get_weather', 'get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_time'], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_time'], got ['get_weather', 'get_weather']
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time'] \| token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_time']
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_time'], got 1 ['get_weather']
`parallel_order`	`rand:1:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_time'], got 2 ['get_weather', 'get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13-parallel-calls --strategy one --strategy rand:1:8 --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -8,14 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Berlin"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {},
-      "name": "get_time"
     }
   ]

Strategies: rand:1:8

--- expected
+++ observed
@@ -14,8 +14,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {},
-      "name": "get_time"
     }
   ]

Strategies: nonstream, special, token, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    },
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512/tree/29439f81c2be264d8d393273f99e7db9c0961120. Tags: parallel-calls, empty-arguments, x-v13-compact.

Raw output

[TOOL_CALLS]get_weather[ARGS]{"city": "Paris"}[TOOL_CALLS]get_weather[ARGS]{"city": "Berlin"}[TOOL_CALLS]get_time[ARGS]{}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    },
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v13-parallel-calls", "family": "mistral", "models": ["mistralai/Ministral-3-14B-Instruct-2512", "mistralai/Devstral-Small-2-24B-Instruct-2512", "mistralai/Magistral-Small-2509"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512/tree/29439f81c2be264d8d393273f99e7db9c0961120", "revision": "29439f81c2be264d8d393273f99e7db9c0961120", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "600bb27946565481ecf51ba8aee252e49b9a68507866080ac9c30185bb312843", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Paris\"}[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Berlin\"}[TOOL_CALLS]get_time[ARGS]{}", "output_token_ids": [9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 42572, 46005, 9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 75613, 46005, 9, 1689, 11332, 32, 30620], "tokenizer": {"repo": "mistralai/Ministral-3-14B-Instruct-2512", "revision": "29439f81c2be264d8d393273f99e7db9c0961120", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Berlin"}}, {"name": "get_time", "arguments": {}}]}, "tags": ["parallel-calls", "empty-arguments", "x-v13-compact"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Ministral-3-14B-Instruct-2512 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Ministral-3-14B-Instruct-2512",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Ministral-3-14B-Instruct-2512",
    "revision": "29439f81c2be264d8d393273f99e7db9c0961120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v13think-content-before-call expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for mistral/v13think-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather'], got []
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['get_weather'] \| token (fail): tool_calls: one [], got ['get_weather'] \| rand:1:8 (fail): tool_calls: one [], got ['get_weather'] \| rand:2:8 (fail): tool_calls: one [], got ['get_weather'] \| rand:3:8 (fail): tool_calls: one [], got ['get_weather'] \| rand:4:8 (fail): tool_calls: one [], got ['get_weather'] \| rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-content-before-call --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": "Let me check that for you.",
   "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check that for you.",
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: single-call, reasoning, text-before-call, x-v13-compact, x-think-special-token.

Raw output

[THINK]The user wants the weather in Paris. I should call get_weather.[/THINK]Let me check that for you.[TOOL_CALLS]get_weather[ARGS]{"city": "Paris"}

Expected parse

{
  "content": "Let me check that for you.",
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v13think-content-before-call", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]The user wants the weather in Paris. I should call get_weather.[/THINK]Let me check that for you.[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Paris\"}", "output_token_ids": [34, 1784, 3330, 17000, 1278, 17253, 1294, 6993, 1046, 1362, 2715, 3690, 2012, 1095, 45629, 1046, 35, 12598, 1639, 4206, 1455, 1394, 1636, 1046, 9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 42572, 46005], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": "Let me check that for you.", "reasoning_content": "The user wants the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "reasoning", "text-before-call", "x-v13-compact", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Magistral-Small-2509",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "MistralDetector",
  "reasoning_effort": "medium",
  "reasoning_enabled": true,
  "reasoning_parser": "mistral",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Magistral-Small-2509",
    "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v13think-parallel-calls expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for mistral/v13think-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-parallel-calls --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -9,17 +9,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes, x-v13-compact, x-think-special-token.

Raw output

[THINK]I should call the tools.[/THINK][TOOL_CALLS]get_weather[ARGS]{"city": "Zürich", "unit": "c"}[TOOL_CALLS]search[ARGS]{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v13think-parallel-calls", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]I should call the tools.[/THINK][TOOL_CALLS]get_weather[ARGS]{\"city\": \"Zürich\", \"unit\": \"c\"}[TOOL_CALLS]search[ARGS]{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}", "output_token_ids": [34, 1073, 2715, 3690, 1278, 12589, 1046, 35, 9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 1090, 2592, 1521, 1897, 1429, 8979, 2811, 1429, 1099, 46005, 9, 8928, 32, 19227, 5272, 2811, 1429, 3173, 1102, 1337, 25994, 22759, 17931, 1897, 1429, 77277, 2811, 16753, 34933, 2811, 12161, 1097, 1897, 1429, 1098, 31597, 1429, 5914, 2811, 1032, 1051, 2821], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes", "x-v13-compact", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Magistral-Small-2509",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "MistralDetector",
  "reasoning_effort": "medium",
  "reasoning_enabled": true,
  "reasoning_parser": "mistral",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Magistral-Small-2509",
    "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v3-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for mistral/v3-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
`arguments_json`	`one`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-empty-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 28 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: single-call, empty-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_time", "arguments": {}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-empty-arguments", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_time\", \"arguments\": {}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 2304, 1316, 1113, 17452, 2032, 1139, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v3-parallel-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/v3-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`token`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`rand:1:8`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`rand:2:8`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`rand:3:8`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`rand:4:8`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`rand:5:8`	fail	content: expected None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}' \| rand:1:8 (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}' \| rand:2:8 (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}' \| rand:3:8 (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}' \| rand:4:8 (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}' \| rand:5:8 (fail): content: one None, got ', {"name": "search", "arguments": {"query": "café \\"best\\"", "filters": {"tags": ["a"}}}'; tool_calls[0].arguments: one '', got '{"city": "Zürich"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`rand:1:8`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`rand:2:8`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`rand:3:8`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`rand:4:8`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`rand:5:8`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 27 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,18 +5,7 @@
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": ", {\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\"}}}",
   "reasoning_content": null,
   "tool_calls": [
@@ -8,15 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "tags": [
            "a"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: parallel-calls, unicode, nested-json, string-escapes, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a"]}}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "tags": [
            "a"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-parallel-calls", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\"}}, {\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\"]}}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 10598, 19141, 2032, 1113, 29596, 3123, 1307, 29507, 11549, 10598, 1629, 2032, 1113, 3128, 1316, 1113, 17452, 2032, 10598, 4153, 2032, 1113, 29485, 2783, 29565, 12547, 14289, 5651, 1316, 1113, 17635, 2032, 10598, 13354, 2032, 8135, 29476, 3010, 1743, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a"]}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/v3-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for mistral/v3-single-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 26 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: single-call, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-single-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 10598, 19141, 2032, 1113, 4684, 1046, 1316, 1113, 6074, 2032, 1113, 29485, 29507, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

fail mistral/vllm-v11-content-before-call expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for mistral/vllm-v11-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['add'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['add'], got []
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['add'] \| token (fail): tool_calls: one [], got ['add'] \| rand:1:8 (fail): tool_calls: one [], got ['add'] \| rand:2:8 (fail): tool_calls: one [], got ['add'] \| rand:3:8 (fail): tool_calls: one [], got ['add'] \| rand:4:8 (fail): tool_calls: one [], got ['add'] \| rand:5:8 (fail): tool_calls: one [], got ['add']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-content-before-call --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": "hi",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "hi",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L621-L631. Tags: single-call, text-before-call, x-v11-args, x-content-then-calls-single-delta.

Raw output

hi[TOOL_CALLS]add[ARGS]{"a": 1, "b": 2}

Expected parse

{
  "content": "hi",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L621-L631", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "hi[TOOL_CALLS]add[ARGS]{\"a\": 1, \"b\": 2}", "output_token_ids": [8101, 9, 2603, 32, 19227, 1097, 2811, 1032, 1049, 1044, 1429, 1098, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "hi", "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 1, "b": 2}}]}, "tags": ["single-call", "text-before-call", "x-v11-args", "x-content-then-calls-single-delta"], "notes": "test_extract_tool_calls (content_before_tool_args)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v11-parallel expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for mistral/vllm-v11-parallel
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['add', 'multiply'], got ['add']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['add', 'multiply'], got ['add']
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| token (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| rand:1:8 (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| rand:2:8 (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| rand:3:8 (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| rand:4:8 (fail): tool_calls: one ['add'], got ['add', 'multiply'] \| rand:5:8 (fail): tool_calls: one ['add'], got ['add', 'multiply']
`parallel_order`	`one`	fail	expected 2 calls ['add', 'multiply'], got 1 ['add']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-parallel --strategy one --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -9,11 +9,4 @@
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "a": 3,
-        "b": 6
-      },
-      "name": "multiply"
     }
   ]

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "a": 3,
        "b": 6
      },
      "name": "multiply"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L605-L620. Tags: parallel-calls, numeric-arguments, x-v11-args.

Raw output

[TOOL_CALLS]add[ARGS]{"a": 3.5, "b": 4}[TOOL_CALLS]multiply[ARGS]{"a": 3, "b": 6}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "a": 3,
        "b": 6
      },
      "name": "multiply"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-parallel", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L605-L620", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}, {"type": "function", "function": {"name": "multiply", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS]add[ARGS]{\"a\": 3.5, \"b\": 4}[TOOL_CALLS]multiply[ARGS]{\"a\": 3, \"b\": 6}", "output_token_ids": [9, 2603, 32, 19227, 1097, 2811, 1032, 1051, 1046, 1053, 1044, 1429, 1098, 2811, 1032, 1052, 1125, 9, 12639, 65975, 32, 19227, 1097, 2811, 1032, 1051, 1044, 1429, 1098, 2811, 1032, 1054, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "multiply", "arguments": {"a": 3, "b": 6}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v11-args"], "notes": "test_extract_tool_calls (multiple_tool_calls_args)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Mistral3ForConditionalGeneration"
    ],
    "model_type": "mistral3"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
    "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail mistral/vllm-v11-think-text expected_match, no_leakage

Checks that did not pass for mistral/vllm-v11-think-text
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`one`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`special`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`token`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`rand:1:8`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`rand:2:8`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`rand:3:8`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`rand:4:8`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`expected_match`	`rand:5:8`	fail	content: expected 'c', got None; reasoning_content: expected 'r', got '<think>r</think>c'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`one`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`special`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`token`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`rand:1:8`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`rand:2:8`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`rand:3:8`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`rand:4:8`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'
`no_leakage`	`rand:5:8`	fail	reasoning_content contains '<think>'; reasoning_content contains '</think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-think-text --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "c",
-  "reasoning_content": "r",
+  "content": null,
+  "reasoning_content": "<think>r</think>c",
   "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_reasoning.py#L286-L301. Tags: no-call, reasoning, x-think-text.

Raw output

<think>r</think>c

Expected parse

{
  "content": "c",
  "reasoning_content": "r",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-think-text", "family": "mistral", "models": ["mistralai/Magistral-Small-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_reasoning.py#L286-L301", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}, "description": "Get the current weather in a city."}}], "raw_output": "<think>r</think>c", "output_token_ids": [49250, 2077, 1062, 1114, 1885, 74045, 1062, 1099], "tokenizer": {"repo": "mistralai/Magistral-Small-2506", "revision": "ad2fce5b4113139e1877dcadfd826c9262ad8e8c", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": "c", "reasoning_content": "r", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-think-text"], "notes": "test_reasoning_v11_plain_text_think: v11 Magistral (2506) reasons inside plain-text <think> tags, which are ordinary tokens, not control tokens."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "6caf38d90087d5c1c405ca0a28fce57478ad0312ab214e0fa9516028e07eaf4a",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Magistral-Small-2506",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      1091,
      1047,
      3174,
      3074,
      1093
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "MistralDetector",
  "reasoning_effort": "medium",
  "reasoning_enabled": true,
  "reasoning_parser": "mistral",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "MistralCommonBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Magistral-Small-2506",
    "revision": "ad2fce5b4113139e1877dcadfd826c9262ad8e8c",
    "trust_remote_code": false
  },
  "tokenizer_mode": "mistral",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v3-arguments-before-name expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v3-arguments-before-name
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"city": "San Francisco", "state": "CA", "unit": "celsius"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "San Francisco", "state": "CA", "unit": "celsius"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "San Francisco", "state": "CA", "unit": "celsius"}'
`arguments_json`	`one`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:1:8`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:2:8`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:5:8`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_current_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-arguments-before-name --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,7 +5,5 @@
     {
       "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_current_weather"

Strategies: nonstream, token, rand:3:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L384-L397. Tags: single-call, x-v3-json-array, x-key-order.

Raw output

[TOOL_CALLS] [{"arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}, "name": "get_current_weather"}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-arguments-before-name", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L384-L397", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}], "raw_output": "[TOOL_CALLS] [{\"arguments\":{\"city\": \"San Francisco\", \"state\": \"CA\", \"unit\": \"celsius\"}, \"name\": \"get_current_weather\"}]", "output_token_ids": [5, 1501, 7567, 17452, 2032, 7567, 19141, 2032, 1113, 18672, 10454, 1316, 1113, 2795, 2032, 1113, 5962, 1316, 1113, 6074, 2032, 1113, 29485, 1958, 3938, 8474, 1113, 1629, 2032, 1113, 1295, 29498, 3790, 29498, 1537, 1991, 29507, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "San Francisco", "state": "CA", "unit": "celsius"}}]}, "tags": ["single-call", "x-v3-json-array", "x-key-order"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (argument_before_name)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v3-content-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v3-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	content (whitespace only): expected 'Hello', got ' Hello'; tool_calls: expected ['add'], got []
`expected_match`	`special`	fail	content (whitespace only): expected 'Hello', got ' Hello'; tool_calls[0].arguments: expected '{"a": 1, "b": 2}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Hello', got ' Hello'
`stream_equals_nonstream`	`one`	fail	content (whitespace only): nonstream 'Hello', got ' Hello'; tool_calls: nonstream ['add'], got []
`stream_equals_nonstream`	`special`	fail	content (whitespace only): nonstream 'Hello', got ' Hello'; tool_calls[0].arguments: nonstream '{"a": 1, "b": 2}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream 'Hello', got ' Hello'
`split_invariance`	`*`	fail	vs one: special (fail): tool_calls: one [], got ['add'] \| token (fail): tool_calls: one [], got ['add'] \| rand:1:8 (fail): tool_calls: one [], got ['add'] \| rand:2:8 (fail): tool_calls: one [], got ['add'] \| rand:3:8 (fail): tool_calls: one [], got ['add'] \| rand:4:8 (fail): tool_calls: one [], got ['add'] \| rand:5:8 (fail): tool_calls: one [], got ['add']
`arguments_json`	`special`	fail	[0] add: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`special`	fail	[0] add: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-content-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": "Hello",
+  "content": " Hello",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": "Hello",
+  "content": " Hello",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "add"

Strategies: nonstream

Matches the expected parse.

{
  "content": "Hello",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Hello",
+  "content": " Hello",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443. Tags: single-call, text-before-call, x-v3-json-array.

Raw output

Hello[TOOL_CALLS] [{"name": "add", "arguments":{"a": 1, "b": 2}}]

Expected parse

{
  "content": "Hello",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "Hello[TOOL_CALLS] [{\"name\": \"add\", \"arguments\":{\"a\": 1, \"b\": 2}}]", "output_token_ids": [23325, 5, 1501, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 7567, 29476, 2032, 29473, 29508, 29493, 1113, 29494, 2032, 29473, 29518, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": "Hello", "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 1, "b": 2}}]}, "tags": ["single-call", "text-before-call", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (content_before_tool)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v3-name-inside-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v3-name-inside-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"name": "John Doe"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"name": "John Doe"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"name": "John Doe"}'
`arguments_json`	`one`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:1:8`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:2:8`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:5:8`	fail	[0] get_age: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_age: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_age: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_age: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_age: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_age: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_age: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-name-inside-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "name": "John Doe"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_age"

Strategies: nonstream, token, rand:3:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "name": "John Doe"
      },
      "name": "get_age"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L398-L413. Tags: single-call, x-v3-json-array, x-param-named-name, x-key-order.

Raw output

[TOOL_CALLS] [{"arguments":{"name": "John Doe"}, "name": "get_age"}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "name": "John Doe"
      },
      "name": "get_age"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-name-inside-arguments", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L398-L413", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_age", "parameters": {"type": "object", "properties": {"name": {"type": "string"}}}}}], "raw_output": "[TOOL_CALLS] [{\"arguments\":{\"name\": \"John Doe\"}, \"name\": \"get_age\"}]", "output_token_ids": [5, 1501, 7567, 17452, 2032, 7567, 1629, 2032, 1113, 15732, 3146, 29474, 8474, 1113, 1629, 2032, 1113, 1295, 29498, 1233, 29507, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_age", "arguments": {"name": "John Doe"}}]}, "tags": ["single-call", "x-v3-json-array", "x-param-named-name", "x-key-order"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (argument_before_name_and_name_in_argument)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v3-no-space expected_match

Checks that did not pass for mistral/vllm-v3-no-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['add'], got []
`expected_match`	`one`	fail	tool_calls: expected ['add'], got []
`expected_match`	`special`	fail	tool_calls: expected ['add'], got []
`expected_match`	`token`	fail	tool_calls: expected ['add'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['add'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['add'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['add'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['add'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['add'], got []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-no-space --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 3.5,
-        "b": 4
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L359-L369. Tags: single-call, numeric-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS][{"name": "add", "arguments":{"a": 3.5, "b": 4}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-no-space", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L359-L369", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS][{\"name\": \"add\", \"arguments\":{\"a\": 3.5, \"b\": 4}}]", "output_token_ids": [5, 29560, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 7567, 29476, 2032, 29473, 29538, 29491, 29550, 29493, 1113, 29494, 2032, 29473, 29549, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}]}, "tags": ["single-call", "numeric-arguments", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (single_tool_add)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail mistral/vllm-v3-parallel expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/vllm-v3-parallel
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`special`	fail	tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`token`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`rand:1:8`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`rand:2:8`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`rand:3:8`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`rand:4:8`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`expected_match`	`rand:5:8`	fail	content: expected None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: expected ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls: nonstream ['add', 'get_current_weather'], got ['add']
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}' \| rand:1:8 (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}' \| rand:2:8 (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}' \| rand:3:8 (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}' \| rand:4:8 (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}' \| rand:5:8 (fail): content: one None, got ', {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}'; tool_calls[0].arguments: one '', got '{"a": 3.5, "b": 4}'
`arguments_json`	`one`	fail	[0] add: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] add: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] add: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] add: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`special`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`token`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`rand:1:8`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`rand:2:8`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`rand:3:8`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`rand:4:8`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']
`parallel_order`	`rand:5:8`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-parallel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,16 +5,7 @@
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
     }
   ]

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": ", {\"name\": \"get_current_weather\", \"arguments\":{\"city\": \"San Francisco\", \"state\": \"CA\", \"unit\": \"celsius\"}}",
   "reasoning_content": null,
   "tool_calls": [
@@ -9,12 +9,4 @@
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L414-L432. Tags: parallel-calls, numeric-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-parallel", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L414-L432", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}, {"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"add\", \"arguments\": {\"a\": 3.5, \"b\": 4}}, {\"name\": \"get_current_weather\", \"arguments\":{\"city\": \"San Francisco\", \"state\": \"CA\", \"unit\": \"celsius\"}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 10598, 29476, 2032, 29473, 29538, 29491, 29550, 29493, 1113, 29494, 2032, 29473, 29549, 11549, 10598, 1629, 2032, 1113, 1295, 29498, 3790, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 7567, 19141, 2032, 1113, 18672, 10454, 1316, 1113, 2795, 2032, 1113, 5962, 1316, 1113, 6074, 2032, 1113, 29485, 1958, 3938, 29507, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "get_current_weather", "arguments": {"city": "San Francisco", "state": "CA", "unit": "celsius"}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (multiple_tools)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

soft pass mistral/v3-no-call expected_match

Checks that did not pass for mistral/v3-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'The capital of France is Paris.', got ' The capital of France is Paris.'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 29 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "The capital of France is Paris.",
+  "content": " The capital of France is Paris.",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: no-call.

Raw output

The capital of France is Paris.

Expected parse

{
  "content": "The capital of France is Paris.",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-no-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "The capital of France is Paris.", "output_token_ids": [1183, 6333, 1070, 5611, 1117, 6233, 29491], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": "The capital of France is Paris.", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "MistralForCausalLM"
    ],
    "model_type": "mistral"
  },
  "hf_config_error": null,
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      4
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 2,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "</s>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "mistralai/Mistral-7B-Instruct-v0.3",
    "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "MistralDetector",
  "tool_call_parser": "mistral",
  "tool_choice": "auto",
  "tools_offered": 6,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model