Qwen3 (Hermes-style JSON tool calls) on vllm 0.30.0

fail 46% strict pass 23 pass · 18 soft pass · 9 fail

Run

Engine

vllm 0.30.0

Run

2026-09-26T05:19:00Z to 2026-09-26T05:20:52Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}

Results file

data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	23	18	2	53%
`expected_error`	3	0	4	43%
`stream_equals_nonstream`	33	8	9	66%
`split_invariance`	50	0	0	100%
`no_leakage`	48	0	2	96%
`arguments_json`	32	0	5	86%
`arguments_schema`	32	0	5	86%
`parallel_order`	7	0	0	100%

Fixtures needing attention

fail qwen3-hermes/bug-sglang-30480-truncated-at-opener stream_equals_nonstream

Checks that did not pass for qwen3-hermes/bug-sglang-30480-truncated-at-opener
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/bug-reports.jsonl --id qwen3-hermes/bug-sglang-30480-truncated-at-opener --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/bug-reports.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "I will check.\n",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "I will check.\n<tool_call>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/30480. Tags: truncated, text-before-call, regression.

Expected graceful failure: Cut by max_tokens right at the <tool_call> opener. The issue reports the Qwen25 detector leaking the bare opener into non-streaming content. (accept: no_tool_calls).

Raw output

I will check.
<tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/bug-sglang-30480-truncated-at-opener", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/30480", "revision": "issue body (2026-07)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "raw_output": "I will check.\n<tool_call>", "output_token_ids": [40, 686, 1779, 624, 151657], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Cut by max_tokens right at the <tool_call> opener. The issue reports the Qwen25 detector leaking the bare opener into non-streaming content.", "accept": ["no_tool_calls"]}, "tags": ["truncated", "text-before-call", "regression"], "notes": "Quoted from the issue: input 'I will check.\\n<tool_call>'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/bug-sglang-30480-truncated-mid-arguments expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/bug-sglang-30480-truncated-mid-arguments
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will check.\n<tool_call>\n{"name": "get_weather", "arguments": {"city": "San Fr', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "San Fr': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/bug-reports.jsonl --id qwen3-hermes/bug-sglang-30480-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/bug-reports.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "I will check.\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"San Fr"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "I will check.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/30480. Tags: truncated, text-before-call, regression.

Expected graceful failure: Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup. (accept: no_tool_calls).

Raw output

I will check.
<tool_call>
{"name": "get_weather", "arguments": {"city": "San Fr

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/bug-sglang-30480-truncated-mid-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/30480", "revision": "issue body (2026-07)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "raw_output": "I will check.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr", "output_token_ids": [40, 686, 1779, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23729, 2869], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.", "accept": ["no_tool_calls"]}, "tags": ["truncated", "text-before-call", "regression"], "notes": "Quoted from the issue: input 'I will check.\\n<tool_call>\\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/marker-in-arguments expected_match, stream_equals_nonstream, no_leakage, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>'; tool_calls: expected ['echo'], got []
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ...'
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>\n{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}}\n</tool_call>', got None; tool_calls: nonstream [], got ['echo']
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'; content contains '<think>'
`arguments_json`	`one`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`special`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`token`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:1:8`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:2:8`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:3:8`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:4:8`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:5:8`	fail	[0] echo: arguments are not valid JSON text ('{"text": "Use <tool_call> ...': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_schema`	`one`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] echo: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>\n{\"name\": \"echo\", \"arguments\": {\"text\": \"Use <tool_call> ... </tool_call> tags, or \\\"<think>\\\" blocks.\"}}\n</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Use <tool_call> ..."
       },
       "name": "echo"

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, marker-in-arguments, string-escapes.

Raw output

<tool_call>
{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/marker-in-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<tool_call>\n{\"name\": \"echo\", \"arguments\": {\"text\": \"Use <tool_call> ... </tool_call> tags, or \\\"<think>\\\" blocks.\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 3047, 497, 330, 16370, 788, 5212, 1318, 788, 330, 10253, 220, 151657, 2503, 220, 151658, 9492, 11, 476, 7245, 151667, 2105, 10010, 1189, 11248, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."}}]}, "tags": ["single-call", "marker-in-arguments", "string-escapes"], "notes": "The argument string contains the literal text of the format markers. The official tokenizer maps them to their added-token ids (151657/151658/151667) inside the JSON string; a correct parser must not end the call at the inner </tool_call>, because the JSON is still open there. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/sglang-malformed-json-in-tags stream_equals_nonstream

Checks that did not pass for qwen3-hermes/sglang-malformed-json-in-tags
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/malformed.jsonl --id qwen3-hermes/sglang-malformed-json-in-tags --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<tool_call>not valid json</tool_call>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L107. Tags: malformed.

Expected graceful failure: The tool_call body is not JSON. SGLang documents returning the original text as content. (accept: no_tool_calls, content_passthrough).

Raw output

<tool_call>not valid json</tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-malformed-json-in-tags", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L107", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>not valid json</tool_call>", "output_token_ids": [151657, 1921, 2697, 2951, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "The tool_call body is not JSON. SGLang documents returning the original text as content.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/truncated-after-open-tag stream_equals_nonstream

Checks that did not pass for qwen3-hermes/truncated-after-open-tag
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>\n', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>\n', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-after-open-tag --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<tool_call>\n",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated.

Expected graceful failure: Output cut by max_tokens right after <tool_call>: there is no call name or arguments. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-after-open-tag", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n", "output_token_ids": [151657, 198], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens right after <tool_call>: there is no call name or arguments.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after '<tool_call>\\n' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/truncated-mid-arguments-no-thinking expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-no-thinking
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Paris', got None; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Paris': Unterminated string starting at: line 1 column 10 (char 9))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after '\"city\": \"Par' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/truncated-mid-arguments-thinking expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-thinking
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`stream_equals_nonstream`	`one`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '\n\n<tool_call>\n{"name": "get_weather", "arguments": {"city', got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city': Unterminated string starting at: line 1 column 2 (char 1))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "\n\n",
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city",
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, reasoning.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<think>
I should call the tool.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tool.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 5392, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning"], "notes": "Token prefix of the full render, cut right after '\"arguments\": {\"ci' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/truncated-second-parallel-call expected_match, stream_equals_nonstream, no_leakage, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>\n{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}\n</tool_call>\n<tool_call>\n{"name": "search", "arguments": {"query": "café', got None; tool_calls: nonstream [], got ['get_weather', 'search']
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`arguments_json`	`one`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`special`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`token`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`rand:1:8`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`rand:2:8`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`rand:3:8`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`rand:4:8`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_json`	`rand:5:8`	fail	[1] search: arguments are not valid JSON text ('{"query": "café': Unterminated string starting at: line 1 column 11 (char 10))
`arguments_schema`	`one`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[1] search: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/qwen3-hermes/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café"
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, parallel-calls, unicode.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-second-parallel-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut mid-arguments. A correct parser keeps the complete call and drops the unterminated one without leaking its markup (the fix proposed in https://github.com/sgl-project/sglang/issues/30480: valid calls survive a truncated neighbour). Token prefix of the full render, cut right after '\"query\": \"caf' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

fail qwen3-hermes/vllm-invalid-json-missing-brace expected_error, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/vllm-invalid-json-missing-brace
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>\n{"name": "final_answer", "arguments": {"trigger": true}', got None; tool_calls: nonstream [], got ['final_answer']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/malformed.jsonl --id qwen3-hermes/vllm-invalid-json-missing-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "trigger": true
      },
      "name": "final_answer"
    }
  ]
}

Strategies: nonstream

{
  "content": "<tool_call>\n{\"name\": \"final_answer\", \"arguments\": {\"trigger\": true}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381. Tags: malformed, truncated.

Expected graceful failure: The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<tool_call>
{"name": "final_answer", "arguments": {"trigger": true}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-invalid-json-missing-brace", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "final_answer", "parameters": {"type": "object", "properties": {"trigger": {"type": "boolean"}}, "required": ["trigger"]}}}], "raw_output": "<tool_call>\n{\"name\": \"final_answer\", \"arguments\": {\"trigger\": true}", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 11822, 28534, 497, 330, 16370, 788, 5212, 18468, 788, 830, 92], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated"], "notes": "vLLM asserts tools_called is false for this output. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/empty-arguments-thinking expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/empty-arguments-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/empty-arguments-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "\nget_time takes no arguments.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "content": "\n\n",
+  "reasoning_content": "\nget_time takes no arguments.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, empty-arguments, reasoning.

Raw output

<think>
get_time takes no arguments.
</think>

<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/empty-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>\nget_time takes no arguments.\n</think>\n\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>", "output_token_ids": [151667, 198, 455, 3009, 4990, 902, 5977, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "get_time takes no arguments.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/instruct2507-text-before-call expected_match

Checks that did not pass for qwen3-hermes/instruct2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "model_rule": "Qwen3-.*Instruct-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Instruct: non-thinking model; docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Instruct-2507",
    "requested_mode": "auto",
    "resolved_revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/multi-turn-final-answer-thinking expected_match

Checks that did not pass for qwen3-hermes/multi-turn-final-answer-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-final-answer-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "It is 20 °C in Zürich right now.",
-  "reasoning_content": "The tool returned 20C.",
+  "content": "\n\nIt is 20 °C in Zürich right now.",
+  "reasoning_content": "\nThe tool returned 20C.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, multi-turn, reasoning, unicode.

Raw output

<think>
The tool returned 20C.
</think>

It is 20 °C in Zürich right now.

Expected parse

{
  "content": "It is 20 °C in Zürich right now.",
  "reasoning_content": "The tool returned 20C.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-final-answer-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe tool returned 20C.\n</think>\n\nIt is 20 °C in Zürich right now.", "output_token_ids": [151667, 198, 785, 5392, 5927, 220, 17, 15, 34, 624, 151668, 271, 2132, 374, 220, 17, 15, 36286, 34, 304, 1863, 5186, 713, 1290, 1431, 13], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "It is 20 °C in Zürich right now.", "reasoning_content": "The tool returned 20C.", "tool_calls": []}, "tags": ["no-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/multi-turn-second-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/multi-turn-second-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-second-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "reasoning_content": "\nIt is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "content": "\n\n",
+  "reasoning_content": "\nIt is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, unicode.

Raw output

<think>
It is 20C in Zürich. Now search for cafés.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "open_now": true
        },
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-second-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nIt is 20C in Zürich. Now search for cafés.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\", \"filters\": {\"open_now\": true}}}\n</tool_call>", "output_token_ids": [151667, 198, 2132, 374, 220, 17, 15, 34, 304, 1863, 5186, 713, 13, 4695, 2711, 369, 29103, 5397, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 497, 330, 24608, 788, 5212, 2508, 20813, 788, 830, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "It is 20C in Zürich. Now search for cafés.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}]}, "tags": ["single-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/no-call-thinking expected_match

Checks that did not pass for qwen3-hermes/no-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/no-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Hello! How can I help you today?",
-  "reasoning_content": "The user is just greeting me. No tool is needed.",
+  "content": "\n\nHello! How can I help you today?",
+  "reasoning_content": "\nThe user is just greeting me. No tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, reasoning.

Raw output

<think>
The user is just greeting me. No tool is needed.
</think>

Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/no-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user is just greeting me. No tool is needed.\n</think>\n\nHello! How can I help you today?", "output_token_ids": [151667, 198, 785, 1196, 374, 1101, 42113, 752, 13, 2308, 5392, 374, 4362, 624, 151668, 271, 9707, 0, 2585, 646, 358, 1492, 498, 3351, 30], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "The user is just greeting me. No tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/parallel-mixed-tools-text-before-no-thinking expected_match

Checks that did not pass for qwen3-hermes/parallel-mixed-tools-text-before-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`one`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-mixed-tools-text-before-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": "I'll fetch both for you.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, text-before-call, empty-arguments.

Raw output

I'll fetch both for you.
<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>
<tool_call>
{"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": "I'll fetch both for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    },
    {
      "arguments": {
        "city": "Oslo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-mixed-tools-text-before-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I'll fetch both for you.\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Oslo\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [40, 3278, 7807, 2176, 369, 498, 624, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 28867, 385, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I'll fetch both for you.", "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}, {"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}]}, "tags": ["parallel-calls", "text-before-call", "empty-arguments"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/parallel-two-calls-thinking expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/parallel-two-calls-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-two-calls-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "\nI should call the tools.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n",
+  "reasoning_content": "\nI should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<think>
I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-two-calls-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/sglang-text-before-call-with-space expected_match

Checks that did not pass for qwen3-hermes/sglang-text-before-call-with-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`one`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`special`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`token`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'I will check the weather for you.', got 'I will check the weather for you. '

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/sglang-text-before-call-with-space --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I will check the weather for you.",
+  "content": "I will check the weather for you. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86. Tags: single-call, text-before-call, x-no-newlines.

Raw output

I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>

Expected parse

{
  "content": "I will check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-text-before-call-with-space", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}], "raw_output": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>", "output_token_ids": [40, 686, 1779, 279, 9104, 369, 498, 13, 220, 151657, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 52854, 16032, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I will check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-no-newlines"], "notes": "Content is the text before the call; the separating space is whitespace (soft-v1). output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/single-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/single-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/single-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "\nThe user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "\nThe user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, reasoning.

Raw output

<think>
The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/single-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151667, 198, 785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/text-before-call-no-thinking expected_match

Checks that did not pass for qwen3-hermes/text-before-call-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ser\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/text-before-call-thinking expected_match

Checks that did not pass for qwen3-hermes/text-before-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London.\n",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call, reasoning.

Raw output

<think>
I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nI need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/thinking2507-multi-turn expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/thinking2507-multi-turn
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-multi-turn --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "content": "\n\n",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, reasoning-prefilled, unicode.

Raw output

It is 20C. Now the café search.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C. Now the café search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-multi-turn", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "It is 20C. Now the café search.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\"}}\n</tool_call>", "output_token_ids": [2132, 374, 220, 17, 15, 34, 13, 4695, 279, 51950, 2711, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "It is 20C. Now the café search.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "reasoning", "reasoning-prefilled", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "model_rule": "Qwen3-.*Thinking-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Thinking model cards (no opening <think>); docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n",
  "reasoning_parser": "deepseek_r1",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "requested_mode": "auto",
    "resolved_revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/thinking2507-no-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "2 + 2 = 4.",
-  "reasoning_content": "Simple arithmetic, no tool needed.",
+  "content": "\n\n2 + 2 = 4.",
+  "reasoning_content": "Simple arithmetic, no tool needed.\n",
   "tool_calls": []
 }

Fixture

Raw output

Simple arithmetic, no tool needed.
</think>

2 + 2 = 4.

Expected parse

{
  "content": "2 + 2 = 4.",
  "reasoning_content": "Simple arithmetic, no tool needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-no-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Simple arithmetic, no tool needed.\n</think>\n\n2 + 2 = 4.", "output_token_ids": [16374, 34784, 11, 902, 5392, 4362, 624, 151668, 271, 17, 488, 220, 17, 284, 220, 19, 13], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "2 + 2 = 4.", "reasoning_content": "Simple arithmetic, no tool needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "model_rule": "Qwen3-.*Thinking-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Thinking model cards (no opening <think>); docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n",
  "reasoning_parser": "deepseek_r1",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "requested_mode": "auto",
    "resolved_revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/thinking2507-parallel-calls expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/thinking2507-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: parallel-calls, reasoning, reasoning-prefilled, unicode, nested-json, string-escapes.

Raw output

I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-parallel-calls", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "I should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "model_rule": "Qwen3-.*Thinking-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Thinking model cards (no opening <think>); docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n",
  "reasoning_parser": "deepseek_r1",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "requested_mode": "auto",
    "resolved_revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/thinking2507-single-call expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/thinking2507-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Raw output

The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-single-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "The user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "model_rule": "Qwen3-.*Thinking-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Thinking model cards (no opening <think>); docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n",
  "reasoning_parser": "deepseek_r1",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "requested_mode": "auto",
    "resolved_revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/thinking2507-text-before-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London.\n",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, text-before-call, reasoning, reasoning-prefilled.

Raw output

I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n<think>\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "model_rule": "Qwen3-.*Thinking-2507",
  "model_type": "qwen3",
  "parser_source": "Qwen3-2507 Thinking model cards (no opening <think>); docs/formats/qwen3-hermes.md",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n<think>\n",
  "reasoning_parser": "deepseek_r1",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "requested_mode": "auto",
    "resolved_revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/truncated-in-reasoning expected_match

Checks that did not pass for qwen3-hermes/truncated-in-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-in-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/qwen3-hermes/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather",
+  "reasoning_content": "\nThe user wants the current weather",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, reasoning.

Raw output

<think>
The user wants the current weather

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-in-reasoning", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user wants the current weather", "output_token_ids": [151667, 198, 785, 1196, 6801, 279, 1482, 9104], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather", "tool_calls": []}, "tags": ["truncated", "reasoning"], "notes": "Cut inside <think> (no </think> yet): the partial text is reasoning, not content. Token prefix of the full render, cut right after 'current weather' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

soft pass qwen3-hermes/unicode-emoji-thinking expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-hermes/unicode-emoji-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/unicode-emoji-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "reasoning_content": "\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "content": "\n\n",
+  "reasoning_content": "\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, unicode, reasoning.

Raw output

<think>
Translate the mixed-script greeting (日本語, العربية, emoji 🌸).
</think>

<tool_call>
{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/unicode-emoji-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text", "target"]}}}], "raw_output": "<think>\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n</think>\n\n<tool_call>\n{\"name\": \"translate\", \"arguments\": {\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}}\n</tool_call>", "output_token_ids": [151667, 198, 27473, 279, 9519, 65960, 42113, 320, 101059, 102819, 11, 129071, 11, 42365, 11162, 234, 116, 4292, 151668, 271, 151657, 198, 4913, 606, 788, 330, 14045, 497, 330, 16370, 788, 5212, 1318, 788, 330, 89015, 11162, 234, 116, 23364, 126860, 124671, 1959, 94880, 586, 51950, 61804, 102, 378, 235, 145851, 497, 330, 5657, 788, 330, 450, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).", "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|im_start|>assistant\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "Qwen/Qwen3-0.6B",
  "model_rule": "Qwen3",
  "model_type": "qwen3",
  "parser_source": "Qwen3 model card: --tool-call-parser hermes --reasoning-parser qwen3",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "end|>\n<|im_start|>user\nUse the tools to answer.<|im_end|>\n<|im_start|>assistant\n",
  "reasoning_parser": "qwen3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "Qwen/Qwen3-0.6B",
    "requested_mode": "auto",
    "resolved_revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "tokenizer_class": "transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "hermes",
  "units_source": "fixture.output_token_ids"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model