Mistral / Magistral / Ministral / Devstral ([TOOL_CALLS]) on llamacpp a25c9865

fail 60% strict pass 28 pass · 19 fail

Run

Engine

llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)

Run

2026-09-26T05:36:17Z to 2026-09-26T05:37:20Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}

Results file

data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Fail	Strict pass rate
`expected_match`	28	17	62%
`expected_error`	0	2	0%
`stream_equals_nonstream`	36	11	77%
`no_leakage`	45	2	96%
`arguments_json`	21	12	64%
`arguments_schema`	21	12	64%
`parallel_order`	3	5	38%

19 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail mistral/llamacpp-v11-call-id expected_match

Checks that did not pass for mistral/llamacpp-v11-call-id
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 15 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248. Tags: single-call, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-v11-call-id", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6244-L6248", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{\"arg1\": 1}", "output_token_ids": [9, 54332, 35214, 33, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-v11-call-id"], "notes": "v11 FUNC_BRACKET_TAG format with a [CALL_ID]. mistral-common's generation grammar has no [CALL_ID], but the v11 tokenizer defines it and llama.cpp parses it. The id is not part of the expected result; the name must not absorb it."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "53278a826a08c3c0291496ff5dd3fb1560dce4e804a079722171783ccb2c40ed",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/llamacpp-v11-call-id-parallel expected_match, parallel_order

Checks that did not pass for mistral/llamacpp-v11-call-id-parallel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`parallel_order`	`nonstream`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 0 []

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/llamacpp-v11-call-id-parallel --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 16 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,19 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
-    {
-      "arguments": {
-        "arg1": 1,
-        "arg2": 2
-      },
-      "name": "special_function_with_opt"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261. Tags: parallel-calls, numeric-arguments, x-v11-call-id.

Raw output

[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{"arg1": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{"arg1": 1, "arg2": 2}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/llamacpp-v11-call-id-parallel", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6250-L6261", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "[TOOL_CALLS]special_function[CALL_ID]000000001[ARGS]{\"arg1\": 1}[TOOL_CALLS]special_function_with_opt[CALL_ID]000000002[ARGS]{\"arg1\": 1, \"arg2\": 2}", "output_token_ids": [9, 54332, 35214, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1049, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1125, 9, 54332, 35214, 23127, 83228, 33, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1048, 1050, 32, 19227, 2230, 1049, 2811, 1032, 1049, 1044, 1429, 2230, 1050, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v11-call-id"]}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "51dc5a86f31c128f6b6704c9ed606a293baadcabe972c33f78428fe8243aed38",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/v11-content-before-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/v11-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Checking the clock.'; tool_calls: nonstream [], got ['get_time']
`arguments_json`	`token`	fail	[0] get_time: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v11-content-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 24 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
-  "content": "Checking the clock.",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token

--- expected
+++ observed
@@ -1,8 +1,11 @@
 {
   "content": "Checking the clock.",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{"
+      },
       "name": "get_time"
     }

Strategies: rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
   "content": "Checking the clock.",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43. Tags: single-call, text-before-call, empty-arguments, x-v11-args.

Raw output

Checking the clock.[TOOL_CALLS]get_time[ARGS]{}

Expected parse

{
  "content": "Checking the clock.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v11-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "6e2501687ccd0e1f30f36319eaf2b46958b897811e246cd8eb5d385b9e3de7d1", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "Checking the clock.[TOOL_CALLS]get_time[ARGS]{}", "output_token_ids": [109758, 1278, 16475, 1046, 9, 1689, 11332, 32, 30620], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "Checking the clock.", "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "text-before-call", "empty-arguments", "x-v11-args"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-Small-3.2-24B-Instruct-2506 (tokenizer v11); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids; with ids, v11 renders [CALL_ID]<id> after the name. Whether the model generates [CALL_ID] is unverified (docs/formats/mistral.md covers both shapes)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/v11-parallel-calls expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/v11-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:5:8 fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v11-parallel-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 23 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,21 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,20 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43. Tags: parallel-calls, unicode, nested-json, string-escapes, x-v11-args.

Raw output

[TOOL_CALLS]get_weather[ARGS]{"city": "Zürich"}[TOOL_CALLS]search[ARGS]{"query": "café \"best\"", "filters": {"max": 3}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v11-parallel-calls", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "6e2501687ccd0e1f30f36319eaf2b46958b897811e246cd8eb5d385b9e3de7d1", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Zürich\"}[TOOL_CALLS]search[ARGS]{\"query\": \"café \\\"best\\\"\", \"filters\": {\"max\": 3}}", "output_token_ids": [9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 1090, 2592, 1521, 46005, 9, 8928, 32, 19227, 5272, 2811, 1429, 3173, 1102, 1337, 25994, 22759, 17931, 1897, 1429, 77277, 2811, 16753, 5914, 2811, 1032, 1051, 2821], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-v11-args"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-Small-3.2-24B-Instruct-2506 (tokenizer v11); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids; with ids, v11 renders [CALL_ID]<id> after the name. Whether the model generates [CALL_ID] is unverified (docs/formats/mistral.md covers both shapes)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/v11-single-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/v11-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v11-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 22 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_weather"

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43. Tags: single-call, x-v11-args.

Raw output

[TOOL_CALLS]get_weather[ARGS]{"city": "Paris", "unit": "c"}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v11-single-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "6e2501687ccd0e1f30f36319eaf2b46958b897811e246cd8eb5d385b9e3de7d1", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS]get_weather[ARGS]{\"city\": \"Paris\", \"unit\": \"c\"}", "output_token_ids": [9, 1689, 1095, 45629, 32, 19227, 29363, 2811, 1429, 42572, 1897, 1429, 8979, 2811, 1429, 1099, 46005], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-v11-args"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-Small-3.2-24B-Instruct-2506 (tokenizer v11); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids; with ids, v11 renders [CALL_ID]<id> after the name. Whether the model generates [CALL_ID] is unverified (docs/formats/mistral.md covers both shapes)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/v13think-truncated-in-arguments expected_error, arguments_json, arguments_schema

Checks that did not pass for mistral/v13think-truncated-in-arguments
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
`expected_error`	`token`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.
`arguments_json`	`nonstream`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh", "content": "rm -rf /': Unterminated string starting at: line 1 column 38 (char 37))
`arguments_schema`	`nonstream`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v13think-truncated-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 9 of fixtures/mistral/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": "Write the script.",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"path\": \"/tmp/build.sh\", \"content\": \"rm -rf /"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49. Tags: truncated, reasoning, x-v13-compact, x-think-special-token.

Expected graceful failure: Output stopped (max_tokens) inside the arguments JSON; the call is incomplete. (accept: no_tool_calls, content_passthrough, exception).

Raw output

[THINK]Write the script.[/THINK][TOOL_CALLS]write_file[ARGS]{"path": "/tmp/build.sh", "content": "rm -rf /

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v13think-truncated-in-arguments", "family": "mistral", "models": ["mistralai/Magistral-Small-2509", "mistralai/Ministral-3-14B-Reasoning-2512", "mistralai/Mistral-Small-4-119B-2603", "mistralai/Mistral-Medium-3.5-128B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Magistral-Small-2509/tree/a31cc96ab10cf19bc42c628fedf1e359e0853c49", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "05e051a32ae9d6333459da0083d36507116334fc85378321cfb32dfff7187531", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[THINK]Write the script.[/THINK][TOOL_CALLS]write_file[ARGS]{\"path\": \"/tmp/build.sh\", \"content\": \"rm -rf /", "output_token_ids": [34, 18746, 1278, 8466, 1046, 35, 9, 16539, 10289, 32, 19227, 7368, 2811, 10977, 20307, 62936, 12519, 1897, 1429, 5431, 2811, 1429, 3650, 1462, 31601, 1987], "tokenizer": {"repo": "mistralai/Magistral-Small-2509", "revision": "a31cc96ab10cf19bc42c628fedf1e359e0853c49", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected_error": {"reason": "Output stopped (max_tokens) inside the arguments JSON; the call is incomplete.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "x-v13-compact", "x-think-special-token"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Magistral-Small-2509 (tokenizer v13); template_sha256 is the sha256 of the tokenizer file that defines the format."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Magistral-Small-2509@a31cc96ab10cf19bc42c628fedf1e359e0853c49",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Magistral-Small-2509",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[THINK]",
    "[/THINK]",
    "[TOOL_CALLS]",
    "[ARGS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Magistral-Small-2509"
    }
  },
  "template_identical_to": "models/templates/unsloth-mistral-Devstral-Small-2507.jinja",
  "template_path": "mistralai--Magistral-Small-2509.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "039a4710e2aabaa79e85a233fa3daa4fbc030a4a8dc218cb7f34fb0c4b8c13b1",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "[/THINK]"
  ],
  "thinking_start_tag": "[THINK]",
  "tokenizer": "mistralai/Magistral-Small-2509@a31cc96ab10cf19bc42c628fedf1e359e0853c49",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Magistral-Small-2509.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "39a36f117102834770556c8fe77bc8ad6405bf43b9546e158a518b6f05c86b30"
}

fail mistral/v3-empty-arguments expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/v3-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_time']
`arguments_json`	`token`	fail	[0] get_time: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_time: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-empty-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 28 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,8 +1,11 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{"
+      },
       "name": "get_time"
     }

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: single-call, empty-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_time", "arguments": {}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-empty-arguments", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_time\", \"arguments\": {}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 2304, 1316, 1113, 17452, 2032, 1139, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/v3-parallel-calls expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/v3-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich"': Expecting ',' delimiter: line 1 column 18 (char 17))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-parallel-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 27 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,23 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token

--- expected
+++ observed
@@ -1,22 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,22 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zür"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,22 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Z"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,22 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,22 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich"
+        "<arguments_raw, not valid JSON>": "{\"city\": \""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "tags": [
-            "a"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: parallel-calls, unicode, nested-json, string-escapes, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a"]}}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "tags": [
            "a"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-parallel-calls", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\"}}, {\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\"]}}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 10598, 19141, 2032, 1113, 29596, 3123, 1307, 29507, 11549, 10598, 1629, 2032, 1113, 3128, 1316, 1113, 17452, 2032, 10598, 4153, 2032, 1113, 29485, 2783, 29565, 12547, 14289, 5651, 1316, 1113, 17635, 2032, 10598, 13354, 2032, 8135, 29476, 3010, 1743, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a"]}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/v3-single-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/v3-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit": "c"': Expecting ',' delimiter: line 1 column 30 (char 29))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/rendered.jsonl --id mistral/v3-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 26 of fixtures/mistral/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit\": \"c\""
       },
       "name": "get_weather"

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris"
       },
       "name": "get_weather"

Strategies: rand:2:8, rand:3:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit\":"
       },
       "name": "get_weather"

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit\": \"c"
       },
       "name": "get_weather"

Strategies: rand:5:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit\": \""
       },
       "name": "get_weather"

Fixture

Provenance: template_render, https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71. Tags: single-call, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/v3-single-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/tree/c170c708c41dac9275d15a8fff4eca08d52bab71", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/render_mistral.py", "template_sha256": "37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89", "attribution": "Rendered with mistral-common 1.12.0 (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "create_user", "description": "Create a user account.", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "arguments": {"type": "string"}, "id": {"type": "string"}}, "required": ["name"]}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1295, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 10598, 19141, 2032, 1113, 4684, 1046, 1316, 1113, 6074, 2032, 1113, 29485, 29507, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-v3-json-array"], "notes": "Rendered with mistral-common 1.12.0 from mistralai/Mistral-7B-Instruct-v0.3 (tokenizer v3); template_sha256 is the sha256 of the tokenizer file that defines the format. Call ids were suppressed (id='null') and mistral-common's request validation was skipped, because the official encode_chat_completion rejects calls without ids."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "03a3066c2c15d81815322622ce3f2447369af69cc427dcc1e53056c0a8615e47",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v11-content-before-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v11-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'hi'; tool_calls: nonstream [], got ['add']
`arguments_json`	`token`	fail	[0] add: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] add: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-content-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 9 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
-  "content": "hi",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": "hi",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "add"

Strategies: rand:1:8, rand:2:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": "hi",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L621-L631. Tags: single-call, text-before-call, x-v11-args, x-content-then-calls-single-delta.

Raw output

hi[TOOL_CALLS]add[ARGS]{"a": 1, "b": 2}

Expected parse

{
  "content": "hi",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L621-L631", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "hi[TOOL_CALLS]add[ARGS]{\"a\": 1, \"b\": 2}", "output_token_ids": [8101, 9, 2603, 32, 19227, 1097, 2811, 1032, 1049, 1044, 1429, 1098, 2811, 1032, 1050, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": "hi", "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 1, "b": 2}}]}, "tags": ["single-call", "text-before-call", "x-v11-args", "x-content-then-calls-single-delta"], "notes": "test_extract_tool_calls (content_before_tool_args)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "d700d48296ecc587a15ba84c2a9e81b33dd972a23fcf264f13d95e29a91fe040",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/vllm-v11-parallel expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/vllm-v11-parallel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['add']
`arguments_json`	`token`	fail	[0] add: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] add: arguments are not a JSON object; not validated
`parallel_order`	`nonstream`	fail	expected 2 calls ['add', 'multiply'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['add', 'multiply'], got 1 ['add']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-parallel --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 8 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,20 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 3.5,
-        "b": 4
-      },
-      "name": "add"
-    },
-    {
-      "arguments": {
-        "a": 3,
-        "b": 6
-      },
-      "name": "multiply"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8

--- expected
+++ observed
@@ -1,19 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "a": 3,
-        "b": 6
-      },
-      "name": "multiply"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L605-L620. Tags: parallel-calls, numeric-arguments, x-v11-args.

Raw output

[TOOL_CALLS]add[ARGS]{"a": 3.5, "b": 4}[TOOL_CALLS]multiply[ARGS]{"a": 3, "b": 6}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "a": 3,
        "b": 6
      },
      "name": "multiply"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-parallel", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L605-L620", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}, {"type": "function", "function": {"name": "multiply", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS]add[ARGS]{\"a\": 3.5, \"b\": 4}[TOOL_CALLS]multiply[ARGS]{\"a\": 3, \"b\": 6}", "output_token_ids": [9, 2603, 32, 19227, 1097, 2811, 1032, 1051, 1046, 1053, 1044, 1429, 1098, 2811, 1032, 1052, 1125, 9, 12639, 65975, 32, 19227, 1097, 2811, 1032, 1051, 1044, 1429, 1098, 2811, 1032, 1054, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "multiply", "arguments": {"a": 3, "b": 6}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v11-args"], "notes": "test_extract_tool_calls (multiple_tool_calls_args)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "716c35582c2e590d90be73c60e9fc7482a5a0b55f84ff222600f7be41f127a37",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/vllm-v11-single-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v11-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['add_this_and_that']
`arguments_json`	`token`	fail	[0] add_this_and_that: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] add_this_and_that: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 3.5,
-        "b": 4
-      },
-      "name": "add_this_and_that"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:3:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "add_this_and_that"

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L578-L590. Tags: single-call, numeric-arguments, x-v11-args.

Raw output

[TOOL_CALLS]add_this_and_that[ARGS]{"a": 3.5, "b": 4}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add_this_and_that"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-single-call", "family": "mistral", "models": ["mistralai/Mistral-Small-3.2-24B-Instruct-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L578-L590", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add_this_and_that", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS]add_this_and_that[ARGS]{\"a\": 3.5, \"b\": 4}", "output_token_ids": [9, 2603, 104922, 27770, 1095, 13800, 32, 19227, 1097, 2811, 1032, 1051, 1046, 1053, 1044, 1429, 1098, 2811, 1032, 1052, 1125], "tokenizer": {"repo": "mistralai/Mistral-Small-3.2-24B-Instruct-2506", "revision": "95a6d26c4bfb886c58daf9d3f7332c857cb27b43", "mode": "mistral"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add_this_and_that", "arguments": {"a": 3.5, "b": 4}}]}, "tags": ["single-call", "numeric-arguments", "x-v11-args"], "notes": "test_extract_tool_calls (single_tool_add_args)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
      "sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-Small-3.2-24B-Instruct-2506@95a6d26c4bfb886c58daf9d3f7332c857cb27b43",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "073a2c733c028526e7e33f85d9adb05a6b2c93bf300f7e6b9ac594d8cf5c3314",
  "vocab_gguf": "mistralai--Mistral-Small-3.2-24B-Instruct-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "473f06dba55a58b6c91dbc5176d0af66d8930a36a4aba0e6391385e913437e77"
}

fail mistral/vllm-v11-think-text expected_match, no_leakage

Checks that did not pass for mistral/vllm-v11-think-text
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'c', got '<think>r</think>c'; reasoning_content: expected 'r', got None
`expected_match`	`token`	fail	content: expected 'c', got '<think>r</think>c'; reasoning_content: expected 'r', got None
`no_leakage`	`nonstream`	fail	content contains '<think>'; content contains '</think>'
`no_leakage`	`token`	fail	content contains '<think>'; content contains '</think>'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v11-think-text --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 11 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "c",
-  "reasoning_content": "r",
+  "content": "<think>r</think>c",
+  "reasoning_content": null,
   "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_reasoning.py#L286-L301. Tags: no-call, reasoning, x-think-text.

Raw output

<think>r</think>c

Expected parse

{
  "content": "c",
  "reasoning_content": "r",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v11-think-text", "family": "mistral", "models": ["mistralai/Magistral-Small-2506"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_reasoning.py#L286-L301", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}, "description": "Get the current weather in a city."}}], "raw_output": "<think>r</think>c", "output_token_ids": [49250, 2077, 1062, 1114, 1885, 74045, 1062, 1099], "tokenizer": {"repo": "mistralai/Magistral-Small-2506", "revision": "ad2fce5b4113139e1877dcadfd826c9262ad8e8c", "mode": "mistral"}, "generation_prompt": "[/INST]", "thinking": true, "expected": {"content": "c", "reasoning_content": "r", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-think-text"], "notes": "test_reasoning_v11_plain_text_think: v11 Magistral (2506) reasons inside plain-text <think> tags, which are ordinary tokens, not control tokens."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Magistral-Small-2506@ad2fce5b4113139e1877dcadfd826c9262ad8e8c",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Magistral-Small-2506",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]",
    "[ARGS]",
    "[CALL_ID]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Magistral-Small-2506"
    }
  },
  "template_identical_to": "models/templates/Mistral-Small-3.2-24B-Instruct-2506.jinja",
  "template_path": "mistralai--Magistral-Small-2506.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "5b8f31f11198696548d544dbe424a3c1ef244d00a475f4e1bffb351fda085608",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Magistral-Small-2506@ad2fce5b4113139e1877dcadfd826c9262ad8e8c",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "c2705ef5e55cbf83d7d702e23105e12caa65966c3ad08d0c04c39943f9af26af",
  "vocab_gguf": "mistralai--Magistral-Small-2506.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "5191ff328c445dfbf97d06c2487231f640c4c44d560ac58c05234e44a6d004f6"
}

fail mistral/vllm-v3-arguments-before-name expected_match

Checks that did not pass for mistral/vllm-v3-arguments-before-name
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-arguments-before-name --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,14 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L384-L397. Tags: single-call, x-v3-json-array, x-key-order.

Raw output

[TOOL_CALLS] [{"arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}, "name": "get_current_weather"}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-arguments-before-name", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L384-L397", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}], "raw_output": "[TOOL_CALLS] [{\"arguments\":{\"city\": \"San Francisco\", \"state\": \"CA\", \"unit\": \"celsius\"}, \"name\": \"get_current_weather\"}]", "output_token_ids": [5, 1501, 7567, 17452, 2032, 7567, 19141, 2032, 1113, 18672, 10454, 1316, 1113, 2795, 2032, 1113, 5962, 1316, 1113, 6074, 2032, 1113, 29485, 1958, 3938, 8474, 1113, 1629, 2032, 1113, 1295, 29498, 3790, 29498, 1537, 1991, 29507, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "San Francisco", "state": "CA", "unit": "celsius"}}]}, "tags": ["single-call", "x-v3-json-array", "x-key-order"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (argument_before_name)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "a1ebc20d717de500edc733770c56c6ec6fd652da03f304c6df554a5845deb174",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v3-content-before-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for mistral/vllm-v3-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Hello'; tool_calls: nonstream [], got ['add']
`arguments_json`	`token`	fail	[0] add: arguments are not valid JSON text ('{"a": 1, "b": 2': Expecting ',' delimiter: line 1 column 16 (char 15))
`arguments_schema`	`token`	fail	[0] add: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-content-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
-  "content": "Hello",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": "Hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 1,
-        "b": 2
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": "Hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": "{\"a\": 1, \"b\": 2"
       },
       "name": "add"

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": "Hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": "{\"a\": 1, \"b"
       },
       "name": "add"

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": "Hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": "{\"a\": 1, \""
       },
       "name": "add"

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,10 +1,10 @@
 {
   "content": "Hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 1,
-        "b": 2
+        "<arguments_raw, not valid JSON>": "{\"a\": 1,"
       },
       "name": "add"

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443. Tags: single-call, text-before-call, x-v3-json-array.

Raw output

Hello[TOOL_CALLS] [{"name": "add", "arguments":{"a": 1, "b": 2}}]

Expected parse

{
  "content": "Hello",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 1,
        "b": 2
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-content-before-call", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L433-L443", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "Hello[TOOL_CALLS] [{\"name\": \"add\", \"arguments\":{\"a\": 1, \"b\": 2}}]", "output_token_ids": [23325, 5, 1501, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 7567, 29476, 2032, 29473, 29508, 29493, 1113, 29494, 2032, 29473, 29518, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": "Hello", "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 1, "b": 2}}]}, "tags": ["single-call", "text-before-call", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (content_before_tool)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "d700d48296ecc587a15ba84c2a9e81b33dd972a23fcf264f13d95e29a91fe040",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v3-malformed-not-json expected_error

Checks that did not pass for mistral/vllm-v3-malformed-not-json
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
`expected_error`	`token`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-malformed-not-json --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/mistral/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L508-L517. Tags: malformed, no-call, x-v3-json-array.

Expected graceful failure: [TOOL_CALLS] is followed by text that is not a JSON array, so there is no tool call. vLLM returns the text after the marker as content. (accept: no_tool_calls, content_passthrough).

Raw output

[TOOL_CALLS] not json at all

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-malformed-not-json", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L508-L517", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS] not json at all", "output_token_ids": [5, 1227, 8379, 1206, 1312], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected_error": {"reason": "[TOOL_CALLS] is followed by text that is not a JSON array, so there is no tool call. vLLM returns the text after the marker as content.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "no-call", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_regex_fallback_fails."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "d700d48296ecc587a15ba84c2a9e81b33dd972a23fcf264f13d95e29a91fe040",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v3-name-inside-arguments expected_match

Checks that did not pass for mistral/vllm-v3-name-inside-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-name-inside-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "name": "John Doe"
-      },
-      "name": "get_age"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L398-L413. Tags: single-call, x-v3-json-array, x-param-named-name, x-key-order.

Raw output

[TOOL_CALLS] [{"arguments":{"name": "John Doe"}, "name": "get_age"}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "name": "John Doe"
      },
      "name": "get_age"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-name-inside-arguments", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L398-L413", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_age", "parameters": {"type": "object", "properties": {"name": {"type": "string"}}}}}], "raw_output": "[TOOL_CALLS] [{\"arguments\":{\"name\": \"John Doe\"}, \"name\": \"get_age\"}]", "output_token_ids": [5, 1501, 7567, 17452, 2032, 7567, 1629, 2032, 1113, 15732, 3146, 29474, 8474, 1113, 1629, 2032, 1113, 1295, 29498, 1233, 29507, 10925], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_age", "arguments": {"name": "John Doe"}}]}, "tags": ["single-call", "x-v3-json-array", "x-param-named-name", "x-key-order"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (argument_before_name_and_name_in_argument)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "b52e76edf2fd423d72cc47137562cf6049ab168d7ceb99c3c452411120ae1119",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v3-no-space expected_match, no_leakage

Checks that did not pass for mistral/vllm-v3-no-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '[TOOL_CALLS][{"name": "add", "arguments":{"a": 3.5, "b": 4}}]'; tool_calls: expected ['add'], got []
`expected_match`	`token`	fail	content: expected None, got '[TOOL_CALLS][{"name": "add", "arguments":{"a": 3.5, "b": 4}}]'; tool_calls: expected ['add'], got []
`no_leakage`	`nonstream`	fail	content contains '[TOOL_CALLS]'
`no_leakage`	`token`	fail	content contains '[TOOL_CALLS]'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-no-space --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "[TOOL_CALLS][{\"name\": \"add\", \"arguments\":{\"a\": 3.5, \"b\": 4}}]",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 3.5,
-        "b": 4
-      },
-      "name": "add"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L359-L369. Tags: single-call, numeric-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS][{"name": "add", "arguments":{"a": 3.5, "b": 4}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-no-space", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L359-L369", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}], "raw_output": "[TOOL_CALLS][{\"name\": \"add\", \"arguments\":{\"a\": 3.5, \"b\": 4}}]", "output_token_ids": [5, 29560, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 7567, 29476, 2032, 29473, 29538, 29491, 29550, 29493, 1113, 29494, 2032, 29473, 29549, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}]}, "tags": ["single-call", "numeric-arguments", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (single_tool_add)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "d700d48296ecc587a15ba84c2a9e81b33dd972a23fcf264f13d95e29a91fe040",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

fail mistral/vllm-v3-parallel expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for mistral/vllm-v3-parallel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['add']
`arguments_json`	`token`	fail	[0] add: arguments are not valid JSON text ('{"a": 3.5, "b": 4': Expecting ',' delimiter: line 1 column 18 (char 17))
`arguments_schema`	`token`	fail	[0] add: arguments are not a JSON object; not validated
`parallel_order`	`nonstream`	fail	expected 2 calls ['add', 'get_current_weather'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['add', 'get_current_weather'], got 1 ['add']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/mistral/imported.jsonl --id mistral/vllm-v3-parallel --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/mistral/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,21 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 3.5,
-        "b": 4
-      },
-      "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:2:8, rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,20 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": "{\"a\": 3.5, \"b\": 4"
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
     }
   ]

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,20 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": "{\"a\": 3.5, \""
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
     }
   ]

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,20 +1,12 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 3.5,
-        "b": 4
+        "<arguments_raw, not valid JSON>": "{\"a\": 3.5"
       },
       "name": "add"
-    },
-    {
-      "arguments": {
-        "city": "San Francisco",
-        "state": "CA",
-        "unit": "celsius"
-      },
-      "name": "get_current_weather"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L414-L432. Tags: parallel-calls, numeric-arguments, x-v3-json-array.

Raw output

[TOOL_CALLS] [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "get_current_weather", "arguments":{"city": "San Francisco", "state": "CA", "unit": "celsius"}}]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 3.5,
        "b": 4
      },
      "name": "add"
    },
    {
      "arguments": {
        "city": "San Francisco",
        "state": "CA",
        "unit": "celsius"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "mistral/vllm-v3-parallel", "family": "mistral", "models": ["mistralai/Mistral-7B-Instruct-v0.3"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/mistral/test_tool_calls.py#L414-L432", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/mistral/import_mistral.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}}}}, {"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}], "raw_output": "[TOOL_CALLS] [{\"name\": \"add\", \"arguments\": {\"a\": 3.5, \"b\": 4}}, {\"name\": \"get_current_weather\", \"arguments\":{\"city\": \"San Francisco\", \"state\": \"CA\", \"unit\": \"celsius\"}}]", "output_token_ids": [5, 1501, 7567, 1629, 2032, 1113, 1756, 1316, 1113, 17452, 2032, 10598, 29476, 2032, 29473, 29538, 29491, 29550, 29493, 1113, 29494, 2032, 29473, 29549, 11549, 10598, 1629, 2032, 1113, 1295, 29498, 3790, 29498, 1537, 1991, 1316, 1113, 17452, 2032, 7567, 19141, 2032, 1113, 18672, 10454, 1316, 1113, 2795, 2032, 1113, 5962, 1316, 1113, 6074, 2032, 1113, 29485, 1958, 3938, 29507, 1743, 29561], "tokenizer": {"repo": "mistralai/Mistral-7B-Instruct-v0.3", "revision": "c170c708c41dac9275d15a8fff4eca08d52bab71", "mode": "hf"}, "generation_prompt": "[/INST]", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, {"name": "get_current_weather", "arguments": {"city": "San Francisco", "state": "CA", "unit": "celsius"}}]}, "tags": ["parallel-calls", "numeric-arguments", "x-v3-json-array"], "notes": "test_extract_tool_calls_pre_v11_tokenizer (multiple_tools)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "</s>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'Ministral/Magistral Large 3', autoparser otherwise",
  "fixture_generation_prompt": "[/INST]",
  "fixture_tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "[TOOL_CALLS]"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after mistralai/Mistral-7B-Instruct-v0.3"
    }
  },
  "template_identical_to": null,
  "template_path": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "mistralai/Mistral-7B-Instruct-v0.3@c170c708c41dac9275d15a8fff4eca08d52bab71",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "ae75ec80821a864b311ebbef5f6d036084d02b235a49f5fde183956abbe882d8",
  "vocab_gguf": "mistralai--Mistral-7B-Instruct-v0.3.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c258ee8aacdec5dd9e8eae9c9869490ebd0e443545e5d3bdd3926c88fe74d6d6"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model