Qwen3 (Hermes-style JSON tool calls) on llamacpp a25c9865

fail 46% strict pass 23 pass · 16 soft pass · 11 fail

Run

Engine

llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)

Run

2026-09-26T05:36:17Z to 2026-09-26T05:37:20Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}

Results file

data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	21	16	6	49%
`expected_error`	3	0	4	43%
`stream_equals_nonstream`	49	0	1	98%
`no_leakage`	45	0	5	90%
`arguments_json`	28	0	4	88%
`arguments_schema`	28	0	4	88%
`parallel_order`	6	0	1	86%

11 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail qwen3-hermes/bug-sglang-30480-truncated-mid-arguments expected_error, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/bug-sglang-30480-truncated-mid-arguments
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/bug-reports.jsonl --id qwen3-hermes/bug-sglang-30480-truncated-mid-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/bug-reports.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "I will check.\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"San Fr"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/30480. Tags: truncated, text-before-call, regression.

Expected graceful failure: Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup. (accept: no_tool_calls).

Raw output

I will check.
<tool_call>
{"name": "get_weather", "arguments": {"city": "San Fr

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/bug-sglang-30480-truncated-mid-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/30480", "revision": "issue body (2026-07)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "raw_output": "I will check.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr", "output_token_ids": [40, 686, 1779, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23729, 2869], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.", "accept": ["no_tool_calls"]}, "tags": ["truncated", "text-before-call", "regression"], "notes": "Quoted from the issue: input 'I will check.\\n<tool_call>\\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "I will check.\n",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"San Fr",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "09331a2e2df9a56a2f3acb61624999968c53eea45d0a0be1bb17cd57f265c61f",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/sglang-text-before-call-with-space expected_match, no_leakage

Checks that did not pass for qwen3-hermes/sglang-text-before-call-with-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'; content contains '</tool_call>'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/sglang-text-before-call-with-space --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "I will check the weather for you.",
+  "content": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86. Tags: single-call, text-before-call, x-no-newlines.

Raw output

I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>

Expected parse

{
  "content": "I will check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-text-before-call-with-space", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}], "raw_output": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>", "output_token_ids": [40, 686, 1779, 279, 9104, 369, 498, 13, 220, 151657, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 52854, 16032, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I will check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-no-newlines"], "notes": "Content is the text before the call; the separating space is whitespace (soft-v1). output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "027d00419b89109c436d15344ab182603b3d5e39b33924a923a3a17157a69f4e",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/truncated-after-open-tag stream_equals_nonstream

Checks that did not pass for qwen3-hermes/truncated-after-open-tag
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n', got None

Synthetic strategies (not counted): stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-after-open-tag --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<tool_call>\n",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated.

Expected graceful failure: Output cut by max_tokens right after <tool_call>: there is no call name or arguments. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-after-open-tag", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n", "output_token_ids": [151657, 198], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens right after <tool_call>: there is no call name or arguments.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after '<tool_call>\\n' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "<tool_call>\n",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/truncated-mid-arguments-no-thinking expected_error, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-no-thinking
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-no-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after '\"city\": \"Par' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Paris",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/truncated-mid-arguments-thinking expected_error, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-thinking
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": "I should call the tool.\n",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, reasoning.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<think>
I should call the tool.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tool.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 5392, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning"], "notes": "Token prefix of the full render, cut right after '\"arguments\": {\"ci' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "I should call the tool.\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"city",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/truncated-second-parallel-call expected_match, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`nonstream`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`token`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_schema`	`nonstream`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[1] search: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-second-parallel-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/qwen3-hermes/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café"
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, parallel-calls, unicode.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-second-parallel-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut mid-arguments. A correct parser keeps the complete call and drops the unterminated one without leaking its markup (the fix proposed in https://github.com/sgl-project/sglang/issues/30480: valid calls survive a truncated neighbour). Token prefix of the full render, cut right after '\"query\": \"caf' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Zürich\", \"unit\": \"c\"}",
            "name": "get_weather"
          },
          {
            "arguments_raw": "{\"query\": \"café",
            "name": "search"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/vllm-content-and-call-single-chunk expected_match, no_leakage

Checks that did not pass for qwen3-hermes/vllm-content-and-call-single-chunk
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>'; tool_calls: expected ['f'], got []
`expected_match`	`token`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>'; tool_calls: expected ['f'], got []
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'; content contains '</tool_call>'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-content-and-call-single-chunk --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Hi!",
+  "content": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "f"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L397. Tags: single-call, text-before-call, numeric-arguments, x-no-newlines.

Raw output

Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>

Expected parse

{
  "content": "Hi!",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "f"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-content-and-call-single-chunk", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L397", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "f", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}}}}}], "raw_output": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}</tool_call>", "output_token_ids": [13048, 0, 151657, 4913, 606, 788, 330, 69, 497, 330, 16370, 788, 5212, 87, 788, 220, 16, 3417, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Hi!", "reasoning_content": null, "tool_calls": [{"name": "f", "arguments": {"x": 1}}]}, "tags": ["single-call", "text-before-call", "numeric-arguments", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}</tool_call>",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "f8c7c5e07da27bf5e1d24f0c8b33a26de90cfd413ea307b6b5dc4a5066e63c73",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/vllm-invalid-json-missing-brace expected_error

Checks that did not pass for qwen3-hermes/vllm-invalid-json-missing-brace
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/malformed.jsonl --id qwen3-hermes/vllm-invalid-json-missing-brace --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "trigger": true
      },
      "name": "final_answer"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381. Tags: malformed, truncated.

Expected graceful failure: The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<tool_call>
{"name": "final_answer", "arguments": {"trigger": true}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-invalid-json-missing-brace", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "final_answer", "parameters": {"type": "object", "properties": {"trigger": {"type": "boolean"}}, "required": ["trigger"]}}}], "raw_output": "<tool_call>\n{\"name\": \"final_answer\", \"arguments\": {\"trigger\": true}", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 11822, 28534, 497, 330, 16370, 788, 5212, 18468, 788, 830, 92], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated"], "notes": "vLLM asserts tools_called is false for this output. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"trigger\": true}",
            "name": "final_answer"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "205862efebceadaad70bfc1e2817a46f4edbd1b0dfa93e3a42f052063f28d98d",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/vllm-no-newlines-no-spaces expected_match, no_leakage

Checks that did not pass for qwen3-hermes/vllm-no-newlines-no-spaces
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`token`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>'; tool_calls: expected ['get_current_temperature'], got []
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'; content contains '</tool_call>'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-no-newlines-no-spaces --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "San Francisco, California, United States",
-        "unit": "celsius"
-      },
-      "name": "get_current_temperature"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L117. Tags: single-call, x-no-newlines.

Raw output

<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "San Francisco, California, United States",
        "unit": "celsius"
      },
      "name": "get_current_temperature"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-no-newlines-no-spaces", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L117", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_current_temperature", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}, "required": ["location"]}}}], "raw_output": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}</tool_call>", "output_token_ids": [151657, 4913, 606, 788, 330, 455, 11080, 53525, 2198, 16370, 788, 5212, 2527, 3252, 23729, 12879, 11, 7043, 11, 3639, 4180, 497, 330, 3843, 788, 330, 66, 40247, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_temperature", "arguments": {"location": "San Francisco, California, United States", "unit": "celsius"}}]}, "tags": ["single-call", "x-no-newlines"], "notes": "No newline inside the tags and irregular JSON spacing; the tags alone delimit the call. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}</tool_call>",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "6848a7bc08b5f00ac367ce9a7d65dbcbe737a6f3b3f37a4aa06465dd88dc4a38",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/vllm-text-then-call-no-separator expected_match, no_leakage

Checks that did not pass for qwen3-hermes/vllm-text-then-call-no-separator
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'; content contains '</tool_call>'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-text-then-call-no-separator --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Sure, let me check the weather.",
+  "content": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "NYC"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L227. Tags: single-call, text-before-call, x-no-newlines.

Raw output

Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>

Expected parse

{
  "content": "Sure, let me check the weather.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "NYC"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-text-then-call-no-separator", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L227", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}</tool_call>", "output_token_ids": [39814, 11, 1077, 752, 1779, 279, 9104, 13, 151657, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23054, 34, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Sure, let me check the weather.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "NYC"}}]}, "tags": ["single-call", "text-before-call", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}</tool_call>",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

fail qwen3-hermes/vllm-two-calls-no-separator expected_match, no_leakage, parallel_order

Checks that did not pass for qwen3-hermes/vllm-two-calls-no-separator
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`token`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>'; tool_calls: expected ['search', 'search'], got []
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`parallel_order`	`nonstream`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['search', 'search'], got 0 []

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-two-calls-no-separator --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,18 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}</tool_call><tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "q": "cats"
-      },
-      "name": "search"
-    },
-    {
-      "arguments": {
-        "q": "dogs"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "q": "cats"
      },
      "name": "search"
    },
    {
      "arguments": {
        "q": "dogs"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-two-calls-no-separator", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L258", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"q": {"type": "string"}}, "required": ["q"]}}}], "raw_output": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}</tool_call><tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}</tool_call>", "output_token_ids": [151657, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 80, 788, 330, 37452, 30975, 151658, 151657, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 80, 788, 330, 80034, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"q": "cats"}}, {"name": "search", "arguments": {"q": "dogs"}}]}, "tags": ["parallel-calls", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}</tool_call><tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}</tool_call>",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "d7f49f78ea8fb594e1aa45f210c0df01f0721e93f3eeb05c17769bd3d0cb111c",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/empty-arguments-thinking expected_match

Checks that did not pass for qwen3-hermes/empty-arguments-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'

Synthetic strategies (not counted): expected_match@one soft pass, expected_match@special soft pass, expected_match@rand:1:8 soft pass, expected_match@rand:2:8 soft pass, expected_match@rand:3:8 soft pass, expected_match@rand:4:8 soft pass, expected_match@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/empty-arguments-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 8 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "get_time takes no arguments.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, empty-arguments, reasoning.

Raw output

<think>
get_time takes no arguments.
</think>

<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/empty-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>\nget_time takes no arguments.\n</think>\n\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>", "output_token_ids": [151667, 198, 455, 3009, 4990, 902, 5977, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "get_time takes no arguments.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "get_time takes no arguments.\n",
        "tool_calls": [
          {
            "arguments_raw": "{}",
            "name": "get_time"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "087e3444fb4f8cb3c090a45e55e6dcb9c26e3c4c39bcf7b3d903f2154627877c",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/instruct2507-text-before-call expected_match

Checks that did not pass for qwen3-hermes/instruct2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Instruct-2507@cdbee75f17c01a7cc42f958dc650907174af0554",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Instruct-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Instruct-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "Qwen/Qwen3-4B-Instruct-2507@cdbee75f17c01a7cc42f958dc650907174af0554",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-4B-Instruct-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "10332d6e429af871611cbcf65ef8d43f660a3020200cff1a0ec5ffbcfb51141a"
}

soft pass qwen3-hermes/multi-turn-final-answer-thinking expected_match

Checks that did not pass for qwen3-hermes/multi-turn-final-answer-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The tool returned 20C.', got 'The tool returned 20C.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The tool returned 20C.', got 'The tool returned 20C.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-final-answer-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "It is 20 °C in Zürich right now.",
-  "reasoning_content": "The tool returned 20C.",
+  "reasoning_content": "The tool returned 20C.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, multi-turn, reasoning, unicode.

Raw output

<think>
The tool returned 20C.
</think>

It is 20 °C in Zürich right now.

Expected parse

{
  "content": "It is 20 °C in Zürich right now.",
  "reasoning_content": "The tool returned 20C.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-final-answer-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe tool returned 20C.\n</think>\n\nIt is 20 °C in Zürich right now.", "output_token_ids": [151667, 198, 785, 5392, 5927, 220, 17, 15, 34, 624, 151668, 271, 2132, 374, 220, 17, 15, 36286, 34, 304, 1863, 5186, 713, 1290, 1431, 13], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "It is 20 °C in Zürich right now.", "reasoning_content": "The tool returned 20C.", "tool_calls": []}, "tags": ["no-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "It is 20 °C in Zürich right now.",
        "exception": null,
        "reasoning_content": "The tool returned 20C.\n",
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/multi-turn-second-call-thinking expected_match

Checks that did not pass for qwen3-hermes/multi-turn-second-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got 'It is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got 'It is 20C in Zürich. Now search for cafés.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-second-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "reasoning_content": "It is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, unicode.

Raw output

<think>
It is 20C in Zürich. Now search for cafés.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "open_now": true
        },
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-second-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nIt is 20C in Zürich. Now search for cafés.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\", \"filters\": {\"open_now\": true}}}\n</tool_call>", "output_token_ids": [151667, 198, 2132, 374, 220, 17, 15, 34, 304, 1863, 5186, 713, 13, 4695, 2711, 369, 29103, 5397, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 497, 330, 24608, 788, 5212, 2508, 20813, 788, 830, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "It is 20C in Zürich. Now search for cafés.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}]}, "tags": ["single-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "It is 20C in Zürich. Now search for cafés.\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"query\": \"café Zürich\", \"filters\": {\"open_now\": true}}",
            "name": "search"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/no-call-thinking expected_match

Checks that did not pass for qwen3-hermes/no-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/no-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Hello! How can I help you today?",
-  "reasoning_content": "The user is just greeting me. No tool is needed.",
+  "reasoning_content": "The user is just greeting me. No tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, reasoning.

Raw output

<think>
The user is just greeting me. No tool is needed.
</think>

Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/no-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user is just greeting me. No tool is needed.\n</think>\n\nHello! How can I help you today?", "output_token_ids": [151667, 198, 785, 1196, 374, 1101, 42113, 752, 13, 2308, 5392, 374, 4362, 624, 151668, 271, 9707, 0, 2585, 646, 358, 1492, 498, 3351, 30], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "The user is just greeting me. No tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Hello! How can I help you today?",
        "exception": null,
        "reasoning_content": "The user is just greeting me. No tool is needed.\n",
        "tool_calls": []
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/parallel-mixed-tools-text-before-no-thinking expected_match

Checks that did not pass for qwen3-hermes/parallel-mixed-tools-text-before-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-mixed-tools-text-before-no-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": "I'll fetch both for you.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, text-before-call, empty-arguments.

Raw output

I'll fetch both for you.
<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>
<tool_call>
{"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": "I'll fetch both for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    },
    {
      "arguments": {
        "city": "Oslo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-mixed-tools-text-before-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I'll fetch both for you.\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Oslo\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [40, 3278, 7807, 2176, 369, 498, 624, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 28867, 385, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I'll fetch both for you.", "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}, {"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}]}, "tags": ["parallel-calls", "text-before-call", "empty-arguments"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "I'll fetch both for you.\n",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{}",
            "name": "get_time"
          },
          {
            "arguments_raw": "{\"city\": \"Oslo\", \"unit\": \"c\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "5f9752061d28dbb2954c6df2e861a68a1a9d66e9d9edb876eb0c6910cf202b29",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/parallel-two-calls-thinking expected_match

Checks that did not pass for qwen3-hermes/parallel-two-calls-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-two-calls-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<think>
I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-two-calls-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "I should call the tools.\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Zürich\", \"unit\": \"c\"}",
            "name": "get_weather"
          },
          {
            "arguments_raw": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}",
            "name": "search"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/single-call-thinking expected_match

Checks that did not pass for qwen3-hermes/single-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/single-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, reasoning.

Raw output

<think>
The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/single-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151667, 198, 785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Paris\", \"unit\": \"c\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/text-before-call-no-thinking expected_match

Checks that did not pass for qwen3-hermes/text-before-call-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-no-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Let me check the current conditions in London.\n",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"London\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/text-before-call-thinking expected_match

Checks that did not pass for qwen3-hermes/text-before-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "Let me check the current conditions in London.\n",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call, reasoning.

Raw output

<think>
I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nI need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Let me check the current conditions in London.\n",
        "exception": null,
        "reasoning_content": "I need live weather data for London.\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"London\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

soft pass qwen3-hermes/thinking2507-multi-turn expected_match

Checks that did not pass for qwen3-hermes/thinking2507-multi-turn
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-multi-turn --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, reasoning-prefilled, unicode.

Raw output

It is 20C. Now the café search.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C. Now the café search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-multi-turn", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "It is 20C. Now the café search.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\"}}\n</tool_call>", "output_token_ids": [2132, 374, 220, 17, 15, 34, 13, 4695, 279, 51950, 2711, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "It is 20C. Now the café search.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "reasoning", "reasoning-prefilled", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Thinking-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0e36cf4a350c75a450c9e1d15b54792f5f5233634ce6c8089863ed7e2ddcfc6b"
}

soft pass qwen3-hermes/thinking2507-no-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-no-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "2 + 2 = 4.",
-  "reasoning_content": "Simple arithmetic, no tool needed.",
+  "reasoning_content": "Simple arithmetic, no tool needed.\n",
   "tool_calls": []
 }

Fixture

Raw output

Simple arithmetic, no tool needed.
</think>

2 + 2 = 4.

Expected parse

{
  "content": "2 + 2 = 4.",
  "reasoning_content": "Simple arithmetic, no tool needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-no-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Simple arithmetic, no tool needed.\n</think>\n\n2 + 2 = 4.", "output_token_ids": [16374, 34784, 11, 902, 5392, 4362, 624, 151668, 271, 17, 488, 220, 17, 284, 220, 19, 13], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "2 + 2 = 4.", "reasoning_content": "Simple arithmetic, no tool needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Thinking-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0e36cf4a350c75a450c9e1d15b54792f5f5233634ce6c8089863ed7e2ddcfc6b"
}

soft pass qwen3-hermes/thinking2507-parallel-calls expected_match

Checks that did not pass for qwen3-hermes/thinking2507-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-parallel-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: parallel-calls, reasoning, reasoning-prefilled, unicode, nested-json, string-escapes.

Raw output

I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-parallel-calls", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "I should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Thinking-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0e36cf4a350c75a450c9e1d15b54792f5f5233634ce6c8089863ed7e2ddcfc6b"
}

soft pass qwen3-hermes/thinking2507-single-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Raw output

The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-single-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "The user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Thinking-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0e36cf4a350c75a450c9e1d15b54792f5f5233634ce6c8089863ed7e2ddcfc6b"
}

soft pass qwen3-hermes/thinking2507-text-before-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "Let me check the current conditions in London.\n",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, text-before-call, reasoning, reasoning-prefilled.

Raw output

I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n<think>\n",
  "fixture_tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after Qwen/Qwen3-4B-Thinking-2507"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-4B-Thinking-2507@768f209d9ea81521153ed38c47d515654e938aea",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "Qwen--Qwen3-4B-Thinking-2507.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0e36cf4a350c75a450c9e1d15b54792f5f5233634ce6c8089863ed7e2ddcfc6b"
}

soft pass qwen3-hermes/unicode-emoji-thinking expected_match

Checks that did not pass for qwen3-hermes/unicode-emoji-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/unicode-emoji-thinking --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, unicode, reasoning.

Raw output

<think>
Translate the mixed-script greeting (日本語, العربية, emoji 🌸).
</think>

<tool_call>
{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/unicode-emoji-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text", "target"]}}}], "raw_output": "<think>\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n</think>\n\n<tool_call>\n{\"name\": \"translate\", \"arguments\": {\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}}\n</tool_call>", "output_token_ids": [151667, 198, 27473, 279, 9519, 65960, 42113, 320, 101059, 102819, 11, 129071, 11, 42365, 11162, 234, 116, 4292, 151668, 271, 151657, 198, 4913, 606, 788, 330, 14045, 497, 330, 16370, 788, 5212, 1318, 788, 330, 89015, 11162, 234, 116, 23364, 126860, 124671, 1959, 94880, 586, 51950, 61804, 102, 378, 235, 145851, 497, 330, 5657, 788, 330, 450, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).", "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|im_start|>assistant\n",
  "fixture_tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "format": "peg-native",
  "generation_prompt": "<|im_start|>assistant\n",
  "model": "Qwen/Qwen3-0.6B",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
        "tool_calls": [
          {
            "arguments_raw": "{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}",
            "name": "translate"
          }
        ]
      },
      "path": "models/templates/Qwen-Qwen3-0.6B.jinja",
      "sha256": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5"
    }
  },
  "template_identical_to": null,
  "template_path": "Qwen--Qwen3-0.6B.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2d05a8253e9a15171a6e24fc30cb4d6a352e4a77b2badcfc02bc757e1b365633",
  "vocab_gguf": "Qwen--Qwen3-0.6B.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "720f095bcaff60b6e960fa8341d2cbb178dcef1bd3ef99e49c8b4e6174022451"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model