Qwen3 (Hermes-style JSON tool calls) on sglang 0.5.20

fail 14% strict pass 7 pass · 4 soft pass · 39 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	6	4	33	14%
`expected_error`	3	0	4	43%
`stream_equals_nonstream`	11	0	39	22%
`split_invariance`	17	0	33	34%
`no_leakage`	44	0	6	88%
`arguments_json`	0	0	32	0%
`arguments_schema`	0	0	32	0%
`parallel_order`	0	0	7	0%

Fixtures needing attention

fail qwen3-hermes/bug-sglang-30480-truncated-at-opener stream_equals_nonstream, split_invariance

Checks that did not pass for qwen3-hermes/bug-sglang-30480-truncated-at-opener
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will check.\n<tool_call>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will check.\n<tool_call>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'I will check.\n<tool_call>', got 'I will check.\n'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will check.\n<tool_call>', got None
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got 'I will check.\n' \| rand:1:8 (fail): content: one None, got 'I will check' \| rand:2:8 (fail): content: one None, got 'I will check' \| rand:3:8 (fail): content: one None, got 'I will check.\n' \| rand:4:8 (fail): content: one None, got 'I will check.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/bug-reports.jsonl --id qwen3-hermes/bug-sglang-30480-truncated-at-opener --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/bug-reports.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: token, rand:3:8, rand:4:8

{
  "content": "I will check.\n",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: rand:1:8, rand:2:8

{
  "content": "I will check",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "I will check.\n<tool_call>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/30480. Tags: truncated, text-before-call, regression.

Expected graceful failure: Cut by max_tokens right at the <tool_call> opener. The issue reports the Qwen25 detector leaking the bare opener into non-streaming content. (accept: no_tool_calls).

Raw output

I will check.
<tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/bug-sglang-30480-truncated-at-opener", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/30480", "revision": "issue body (2026-07)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "raw_output": "I will check.\n<tool_call>", "output_token_ids": [40, 686, 1779, 624, 151657], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Cut by max_tokens right at the <tool_call> opener. The issue reports the Qwen25 detector leaking the bare opener into non-streaming content.", "accept": ["no_tool_calls"]}, "tags": ["truncated", "text-before-call", "regression"], "notes": "Quoted from the issue: input 'I will check.\\n<tool_call>'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/bug-sglang-30480-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/bug-sglang-30480-truncated-mid-arguments
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will check.', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will check.', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream 'I will check.', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will check.', got 'I will check'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will check.', got 'I will check'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content (whitespace only): nonstream 'I will check.', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content (whitespace only): nonstream 'I will check.', got 'I will check.\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will check.', got None; tool_calls: nonstream [], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got 'I will check.\n'; tool_calls[0].arguments: one '', got '{"city": "San Fr"}' \| rand:1:8 (fail): content: one None, got 'I will check'; tool_calls[0].arguments: one '', got '{"city": "San Fr"}' \| rand:2:8 (fail): content: one None, got 'I will check'; tool_calls[0].arguments: one '', got '{"city": "San Fr"}' \| rand:3:8 (fail): content: one None, got 'I will check.\n'; tool_calls[0].arguments: one '', got '{"city": "San Fr"}' \| rand:4:8 (fail): content: one None, got 'I will check.\n'; tool_calls[0].arguments: one '', got '{"city": "San Fr"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "San Fr"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/bug-reports.jsonl --id qwen3-hermes/bug-sglang-30480-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/bug-reports.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:3:8, rand:4:8

{
  "content": "I will check.\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Fr"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: rand:1:8, rand:2:8

{
  "content": "I will check",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Fr"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Fr"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "I will check.",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/30480. Tags: truncated, text-before-call, regression.

Expected graceful failure: Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup. (accept: no_tool_calls).

Raw output

I will check.
<tool_call>
{"name": "get_weather", "arguments": {"city": "San Fr

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/bug-sglang-30480-truncated-mid-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/30480", "revision": "issue body (2026-07)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "raw_output": "I will check.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr", "output_token_ids": [40, 686, 1779, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23729, 2869], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Cut by max_tokens mid-arguments. The issue reports non-streaming leaking the raw <tool_call> markup into content while streaming drops it; a correct parser returns no call and does not leak markup.", "accept": ["no_tool_calls"]}, "tags": ["truncated", "text-before-call", "regression"], "notes": "Quoted from the issue: input 'I will check.\\n<tool_call>\\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"San Fr'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/empty-arguments-no-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/empty-arguments-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
`arguments_json`	`one`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/empty-arguments-no-thinking --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, empty-arguments.

Raw output

<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/empty-arguments-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments"], "notes": "Parameterless tool, arguments {} (cf. https://github.com/vllm-project/vllm/issues/28806). History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/empty-arguments-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/empty-arguments-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'; tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'; tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`expected_match`	`rand:4:8`	fail	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'; tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got '\nget_time takes no arguments.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{}' \| rand:1:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
`arguments_json`	`one`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/empty-arguments-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:4:8

--- expected
+++ observed
@@ -1,8 +1,10 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "\nget_time takes no arguments.\n",
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, rand:2:8, rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "\nget_time takes no arguments.\n",
   "tool_calls": [
     {

Strategies: token, rand:1:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "content": "\n\n",
+  "reasoning_content": "\nget_time takes no arguments.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, empty-arguments, reasoning.

Raw output

<think>
get_time takes no arguments.
</think>

<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/empty-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>\nget_time takes no arguments.\n</think>\n\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>", "output_token_ids": [151667, 198, 455, 3009, 4990, 902, 5977, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "get_time takes no arguments.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/instruct2507-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/instruct2507-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
`arguments_json`	`one`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-empty-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: single-call, empty-arguments.

Raw output

<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-empty-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Instruct-2507",
    "revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/instruct2507-parallel-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/instruct2507-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-parallel-calls --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,21 +5,7 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: parallel-calls, unicode, nested-json, string-escapes.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-parallel-calls", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Instruct-2507",
    "revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/instruct2507-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/instruct2507-single-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: single-call.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-single-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Instruct-2507",
    "revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/instruct2507-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/instruct2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	content: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`special`	fail	content: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`expected_match`	`rand:2:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`expected_match`	`rand:3:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got 'Let me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:1:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:3:8 (fail): content: one None, got 'Let me check the current conditions in'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/instruct-2507.jsonl --id qwen3-hermes/instruct2507-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/instruct-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/instruct2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-30B-A3B-Instruct-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/cdbee75f17c01a7cc42f958dc650907174af0554/tokenizer_config.json", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Instruct-2507", "revision": "cdbee75f17c01a7cc42f958dc650907174af0554", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "64f85b198065d0fba2a81f37e10ed68161ce2c19a754c7100e67e0ca2ee9c326",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Instruct-2507",
    "revision": "cdbee75f17c01a7cc42f958dc650907174af0554",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/json-string-argument expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/json-string-argument
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"text": "{\\"name\\": \\"x\\", \\"arguments\\": {\\"a\\": 1}}"}'
`arguments_json`	`one`	fail	[0] echo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] echo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] echo: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/json-string-argument --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "{\"name\": \"x\", \"arguments\": {\"a\": 1}}"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "echo"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "{\"name\": \"x\", \"arguments\": {\"a\": 1}}"
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, string-escapes, x-json-in-string.

Raw output

<tool_call>
{"name": "echo", "arguments": {"text": "{\"name\": \"x\", \"arguments\": {\"a\": 1}}"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "{\"name\": \"x\", \"arguments\": {\"a\": 1}}"
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/json-string-argument", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<tool_call>\n{\"name\": \"echo\", \"arguments\": {\"text\": \"{\\\"name\\\": \\\"x\\\", \\\"arguments\\\": {\\\"a\\\": 1}}\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 3047, 497, 330, 16370, 788, 5212, 1318, 788, 54734, 606, 11693, 7245, 87, 16215, 7245, 16370, 11693, 314, 2105, 64, 11693, 220, 16, 3417, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "{\"name\": \"x\", \"arguments\": {\"a\": 1}}"}}]}, "tags": ["single-call", "string-escapes", "x-json-in-string"], "notes": "A string argument that itself looks like a Hermes tool-call payload; it must stay a string. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/long-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/long-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d..., got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d..., got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d..., got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d..., got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d... \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d... \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d... \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d... \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d... \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy d...
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/long-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n",
-        "path": "notes/sample.txt"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "write_file"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n",
        "path": "notes/sample.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, long-arguments, unicode.

Raw output

<tool_call>
{"name": "write_file", "arguments": {"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n",
        "path": "notes/sample.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/long-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}], "raw_output": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"notes/sample.txt\", \"content\": \"001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\\n\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 4934, 2458, 497, 330, 16370, 788, 5212, 2343, 788, 330, 18286, 69851, 3909, 497, 330, 1796, 788, 330, 15, 15, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 15, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 16, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 17, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 18, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 19, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 16, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 17, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 18, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 19, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 20, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 21, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 22, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 23, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 20, 24, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 15, 21, 15, 25, 576, 3974, 13876, 38835, 34208, 916, 279, 15678, 5562, 26, 51950, 11, 94880, 586, 11, 60596, 109, 46553, 7110, 77, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "notes/sample.txt", "content": "001: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n002: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n003: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n004: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n005: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n006: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n007: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n008: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n009: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n010: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n011: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n012: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n013: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n014: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n015: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n016: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n017: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n018: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n019: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n020: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n021: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n022: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n023: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n024: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n025: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n026: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n027: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n028: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n029: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n030: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n031: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n032: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n033: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n034: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n035: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n036: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n037: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n038: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n039: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n040: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n041: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n042: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n043: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n044: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n045: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n046: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n047: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n048: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n049: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n050: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n051: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n052: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n053: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n054: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n055: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n056: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n057: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n058: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n059: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n060: The quick brown fox jumps over the lazy dog; café, naïve, 東京.\n"}}]}, "tags": ["single-call", "long-arguments", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got ''
`expected_match`	`token`	fail	reasoning_content: expected None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}'
`expected_match`	`rand:1:8`	fail	reasoning_content: expected None, got '</tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ..."}'
`expected_match`	`rand:2:8`	fail	reasoning_content: expected None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}'
`expected_match`	`rand:3:8`	fail	reasoning_content: expected None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}'
`expected_match`	`rand:4:8`	fail	reasoning_content: expected None, got ' ... </tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call>"}'
`expected_match`	`rand:5:8`	fail	reasoning_content: expected None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: expected '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got ''
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream None, got '</tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ..."}'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream None, got ' ... </tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call>"}'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: nonstream '{"text": "Use <tool_call> ... </tool_call> tags, or \\"<think>\\" blocks."}', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}'
`split_invariance`	`*`	fail	vs one: token (fail): reasoning_content: one None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}' \| rand:1:8 (fail): reasoning_content: one None, got '</tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call> ..."}' \| rand:2:8 (fail): reasoning_content: one None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}' \| rand:3:8 (fail): reasoning_content: one None, got ' \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call> ... </tool_call> tags, or"}' \| rand:4:8 (fail): reasoning_content: one None, got ' ... </tool_call> tags, or \\"\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call>"}' \| rand:5:8 (fail): reasoning_content: one None, got '\\" blocks."}}\n</tool_call>'; tool_calls[0].arguments: one '', got '{"text": "Use <tool_call> ... </tool_call> tags, or \\""}'
`no_leakage`	`token`	fail	reasoning_content contains '</tool_call>'
`no_leakage`	`rand:1:8`	fail	reasoning_content contains '</tool_call>'
`no_leakage`	`rand:2:8`	fail	reasoning_content contains '</tool_call>'
`no_leakage`	`rand:3:8`	fail	reasoning_content contains '</tool_call>'
`no_leakage`	`rand:4:8`	fail	reasoning_content contains '</tool_call>'
`no_leakage`	`rand:5:8`	fail	reasoning_content contains '</tool_call>'
`arguments_json`	`one`	fail	[0] echo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] echo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] echo: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] echo: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "echo"

Strategies: token, rand:5:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
   "content": null,
-  "reasoning_content": null,
+  "reasoning_content": "\\\" blocks.\"}}\n</tool_call>",
   "tool_calls": [
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "text": "Use <tool_call> ... </tool_call> tags, or \""
       },
       "name": "echo"

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
   "content": null,
-  "reasoning_content": null,
+  "reasoning_content": "</tool_call> tags, or \\\"\\\" blocks.\"}}\n</tool_call>",
   "tool_calls": [
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "text": "Use <tool_call> ..."
       },
       "name": "echo"

Strategies: rand:2:8, rand:3:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
   "content": null,
-  "reasoning_content": null,
+  "reasoning_content": " \\\"\\\" blocks.\"}}\n</tool_call>",
   "tool_calls": [
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "text": "Use <tool_call> ... </tool_call> tags, or"
       },
       "name": "echo"

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
   "content": null,
-  "reasoning_content": null,
+  "reasoning_content": " ... </tool_call> tags, or \\\"\\\" blocks.\"}}\n</tool_call>",
   "tool_calls": [
     {
       "arguments": {
-        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
+        "text": "Use <tool_call>"
       },
       "name": "echo"

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, marker-in-arguments, string-escapes.

Raw output

<tool_call>
{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/marker-in-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<tool_call>\n{\"name\": \"echo\", \"arguments\": {\"text\": \"Use <tool_call> ... </tool_call> tags, or \\\"<think>\\\" blocks.\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 3047, 497, 330, 16370, 788, 5212, 1318, 788, 330, 10253, 220, 151657, 2503, 220, 151658, 9492, 11, 476, 7245, 151667, 2105, 10010, 1189, 11248, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Use <tool_call> ... </tool_call> tags, or \"<think>\" blocks."}}]}, "tags": ["single-call", "marker-in-arguments", "string-escapes"], "notes": "The argument string contains the literal text of the format markers. The official tokenizer maps them to their added-token ids (151657/151658/151667) inside the JSON string; a correct parser must not end the call at the inner </tool_call>, because the JSON is still open there. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/multi-turn-second-call-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/multi-turn-second-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'; tool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'; tool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C in Zürich. Now search for cafés.', got '\nIt is 20C in Zürich. Now search for cafés.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"query": "café Zürich", "filters": {"open_now": true}}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"query": "café Zürich", "filters": {"open_now": true}}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}' \| rand:3:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich", "filters": {"open_now": true}}'
`arguments_json`	`one`	fail	[0] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] search: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] search: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-second-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,12 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "reasoning_content": "\nIt is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {
       "arguments": {
-        "filters": {
-          "open_now": true
-        },
-        "query": "café Zürich"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "search"

Strategies: nonstream, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "reasoning_content": "\nIt is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
+  "content": "\n\n",
+  "reasoning_content": "\nIt is 20C in Zürich. Now search for cafés.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, unicode.

Raw output

<think>
It is 20C in Zürich. Now search for cafés.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C in Zürich. Now search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "open_now": true
        },
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-second-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nIt is 20C in Zürich. Now search for cafés.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\", \"filters\": {\"open_now\": true}}}\n</tool_call>", "output_token_ids": [151667, 198, 2132, 374, 220, 17, 15, 34, 304, 1863, 5186, 713, 13, 4695, 2711, 369, 29103, 5397, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 497, 330, 24608, 788, 5212, 2508, 20813, 788, 830, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "It is 20C in Zürich. Now search for cafés.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}]}, "tags": ["single-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/nested-json-deep expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/nested-json-deep
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an... \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an... \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an... \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an... \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an... \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an...
`arguments_json`	`one`	fail	[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] create_event: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/nested-json-deep --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,37 +5,5 @@
     {
       "arguments": {
-        "attendees": [
-          {
-            "email": "ana@example.com",
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "email": "bo@example.com",
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes",
-              "timekeeper"
-            ]
-          }
-        ],
-        "location": null,
-        "reminders": [
-          [
-            10,
-            "email"
-          ],
-          [
-            1,
-            "popup"
-          ]
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z",
-          "tz": "UTC"
-        }
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "create_event"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "email": "ana@example.com",
            "name": "Ana",
            "optional": false
          },
          {
            "email": "bo@example.com",
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes",
              "timekeeper"
            ]
          }
        ],
        "location": null,
        "reminders": [
          [
            10,
            "email"
          ],
          [
            1,
            "popup"
          ]
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z",
          "tz": "UTC"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, nested-json.

Raw output

<tool_call>
{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "ana@example.com", "optional": false}, {"name": "Bo", "email": "bo@example.com", "optional": true, "roles": ["notes", "timekeeper"]}], "reminders": [[10, "email"], [1, "popup"]], "location": null}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "email": "ana@example.com",
            "name": "Ana",
            "optional": false
          },
          {
            "email": "bo@example.com",
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes",
              "timekeeper"
            ]
          }
        ],
        "location": null,
        "reminders": [
          [
            10,
            "email"
          ],
          [
            1,
            "popup"
          ]
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z",
          "tz": "UTC"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/nested-json-deep", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}, "reminders": {"type": "array"}, "location": {"type": ["object", "null"]}}, "required": ["title", "when"]}}}], "raw_output": "<tool_call>\n{\"name\": \"create_event\", \"arguments\": {\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\", \"tz\": \"UTC\"}, \"attendees\": [{\"name\": \"Ana\", \"email\": \"ana@example.com\", \"optional\": false}, {\"name\": \"Bo\", \"email\": \"bo@example.com\", \"optional\": true, \"roles\": [\"notes\", \"timekeeper\"]}], \"reminders\": [[10, \"email\"], [1, \"popup\"]], \"location\": null}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 3182, 6748, 497, 330, 16370, 788, 5212, 2102, 788, 330, 20470, 3395, 497, 330, 9309, 788, 5212, 2468, 788, 330, 17, 15, 17, 21, 12, 16, 15, 12, 15, 16, 51, 15, 24, 25, 15, 15, 25, 15, 15, 57, 497, 330, 408, 788, 330, 17, 15, 17, 21, 12, 16, 15, 12, 15, 16, 51, 16, 15, 25, 15, 15, 25, 15, 15, 57, 497, 330, 37592, 788, 330, 21183, 14345, 330, 56458, 5516, 788, 61753, 606, 788, 330, 80586, 497, 330, 2332, 788, 330, 3362, 35487, 905, 497, 330, 12807, 788, 895, 2137, 5212, 606, 788, 330, 1233, 497, 330, 2332, 788, 330, 749, 35487, 905, 497, 330, 12807, 788, 830, 11, 330, 14643, 788, 4383, 18286, 497, 330, 1678, 18861, 1341, 72441, 330, 1826, 31451, 788, 4318, 16, 15, 11, 330, 2332, 7914, 508, 16, 11, 330, 23706, 1341, 1125, 330, 2527, 788, 845, 11248, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "ana@example.com", "optional": false}, {"name": "Bo", "email": "bo@example.com", "optional": true, "roles": ["notes", "timekeeper"]}], "reminders": [[10, "email"], [1, "popup"]], "location": null}}]}, "tags": ["single-call", "nested-json"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/numeric-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/numeric-arguments
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}'
`arguments_json`	`one`	fail	[0] calculate: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] calculate: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] calculate: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] calculate: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/numeric-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,14 +5,5 @@
     {
       "arguments": {
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "calculate"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, numeric-arguments.

Raw output

<tool_call>
{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/numeric-arguments", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}}, "required": ["x", "y"]}}}], "raw_output": "<tool_call>\n{\"name\": \"calculate\", \"arguments\": {\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21]}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 35597, 497, 330, 16370, 788, 5212, 87, 788, 481, 18, 11, 330, 88, 788, 220, 17, 13, 20, 11, 330, 20576, 261, 681, 788, 220, 16, 68, 12, 15, 20, 11, 330, 46385, 788, 830, 11, 330, 9506, 788, 845, 11, 330, 3661, 788, 508, 15, 11, 481, 15, 13, 20, 11, 220, 16, 68, 10, 17, 16, 60, 11248, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}}]}, "tags": ["single-call", "numeric-arguments"], "notes": "Types matter: integers, floats, exponent notation, booleans and null must round-trip. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/parallel-mixed-tools-text-before-no-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/parallel-mixed-tools-text-before-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	content: expected "I'll fetch both for you.", got None; tool_calls: expected ['get_time', 'get_weather'], got ['get_time']
`expected_match`	`special`	fail	content: expected "I'll fetch both for you.", got None; tool_calls: expected ['get_time', 'get_weather'], got ['get_time']
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:1:8`	fail	content: expected "I'll fetch both for you.", got "I'll fetch both for"
`expected_match`	`rand:2:8`	fail	content: expected "I'll fetch both for you.", got "I'll fetch both for"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll fetch both for you.", got "I'll fetch both for you.\n"
`expected_match`	`rand:4:8`	fail	content: expected "I'll fetch both for you.", got "I'll fetch both"
`expected_match`	`rand:5:8`	fail	content: expected "I'll fetch both for you.", got "I'll fetch both for"
`stream_equals_nonstream`	`one`	fail	content: nonstream "I'll fetch both for you.", got None; tool_calls: nonstream ['get_time', 'get_weather'], got ['get_time']
`stream_equals_nonstream`	`special`	fail	content: nonstream "I'll fetch both for you.", got None; tool_calls: nonstream ['get_time', 'get_weather'], got ['get_time']
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream "I'll fetch both for you.", got "I'll fetch both for you.\n"
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream "I'll fetch both for you.", got "I'll fetch both for"
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream "I'll fetch both for you.", got "I'll fetch both for"
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream "I'll fetch both for you.", got "I'll fetch both for you.\n"
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream "I'll fetch both for you.", got "I'll fetch both"
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream "I'll fetch both for you.", got "I'll fetch both for"
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got "I'll fetch both for you.\n"; tool_calls: one ['get_time'], got ['get_time', 'get_weather'] \| rand:1:8 (fail): content: one None, got "I'll fetch both for"; tool_calls: one ['get_time'], got ['get_time', 'get_weather'] \| rand:2:8 (fail): content: one None, got "I'll fetch both for"; tool_calls: one ['get_time'], got ['get_time', 'get_weather'] \| rand:3:8 (fail): content: one None, got "I'll fetch both for you.\n"; tool_calls: one ['get_time'], got ['get_time', 'get_weather'] \| rand:4:8 (fail): content: one None, got "I'll fetch both"; tool_calls: one ['get_time'], got ['get_time', 'get_weather'] \| rand:5:8 (fail): content: one None, got "I'll fetch both for"; tool_calls: one ['get_time'], got ['get_time', 'get_weather']
`arguments_json`	`one`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_time: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_time: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['get_time', 'get_weather'], got 1 ['get_time']
`parallel_order`	`special`	fail	expected 2 calls ['get_time', 'get_weather'], got 1 ['get_time']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-mixed-tools-text-before-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,16 +1,11 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
-    },
-    {
-      "arguments": {
-        "city": "Oslo",
-        "unit": "c"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": "I'll fetch both for",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": "I'll fetch both",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream

Matches the expected parse.

{
  "content": "I'll fetch both for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    },
    {
      "arguments": {
        "city": "Oslo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll fetch both for you.",
+  "content": "I'll fetch both for you.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, text-before-call, empty-arguments.

Raw output

I'll fetch both for you.
<tool_call>
{"name": "get_time", "arguments": {}}
</tool_call>
<tool_call>
{"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": "I'll fetch both for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    },
    {
      "arguments": {
        "city": "Oslo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-mixed-tools-text-before-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I'll fetch both for you.\n<tool_call>\n{\"name\": \"get_time\", \"arguments\": {}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Oslo\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [40, 3278, 7807, 2176, 369, 498, 624, 151657, 198, 4913, 606, 788, 330, 455, 3009, 497, 330, 16370, 788, 4687, 532, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 28867, 385, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I'll fetch both for you.", "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}, {"name": "get_weather", "arguments": {"city": "Oslo", "unit": "c"}}]}, "tags": ["parallel-calls", "text-before-call", "empty-arguments"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/parallel-three-calls-no-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/parallel-three-calls-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-three-calls-no-thinking --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,18 +5,5 @@
     {
       "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris"}}
</tool_call>
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>
<tool_call>
{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-three-calls-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 52854, 16032, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "London"}}, {"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "c"}}]}, "tags": ["parallel-calls"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/parallel-two-calls-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/parallel-two-calls-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got '\nI should call the tools.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/parallel.jsonl --id qwen3-hermes/parallel-two-calls-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,25 +1,11 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "\nI should call the tools.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "\nI should call the tools.\n",
   "tool_calls": [
     {

Strategies: token, rand:1:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n",
+  "reasoning_content": "\nI should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<think>
I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/parallel-two-calls-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/sglang-malformed-json-in-tags stream_equals_nonstream

Checks that did not pass for qwen3-hermes/sglang-malformed-json-in-tags
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>not valid json</tool_call>', got '<tool_call>not valid json'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/malformed.jsonl --id qwen3-hermes/sglang-malformed-json-in-tags --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "<tool_call>not valid json",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<tool_call>not valid json</tool_call>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L107. Tags: malformed.

Expected graceful failure: The tool_call body is not JSON. SGLang documents returning the original text as content. (accept: no_tool_calls, content_passthrough).

Raw output

<tool_call>not valid json</tool_call>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-malformed-json-in-tags", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L107", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>not valid json</tool_call>", "output_token_ids": [151657, 1921, 2697, 2951, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "The tool_call body is not JSON. SGLang documents returning the original text as content.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/sglang-qwen25-four-parallel-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/sglang-qwen25-four-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got ['get_current_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got ['get_current_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got ['get_current_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got ['get_current_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'] \| rand:1:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'] \| rand:2:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'] \| rand:3:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'] \| rand:4:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'] \| rand:5:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather']
`arguments_json`	`one`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_current_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 4 calls ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got 1 ['get_current_weather']
`parallel_order`	`special`	fail	expected 4 calls ['get_current_weather', 'get_current_weather', 'get_current_weather', 'get_current_weather'], got 1 ['get_current_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/sglang-qwen25-four-parallel-calls --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,31 +5,5 @@
     {
       "arguments": {
-        "city": "NYC",
-        "state": "NY",
-        "unit": "fahrenheit"
-      },
-      "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Baltimore",
-        "state": "MD",
-        "unit": "fahrenheit"
-      },
-      "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Minneapolis",
-        "state": "MN",
-        "unit": "fahrenheit"
-      },
-      "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Los Angeles",
-        "state": "CA",
-        "unit": "fahrenheit"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_current_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "NYC",
        "state": "NY",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Baltimore",
        "state": "MD",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Minneapolis",
        "state": "MN",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Los Angeles",
        "state": "CA",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5416. Tags: parallel-calls.

Raw output

<tool_call>
{"name": "get_current_weather", "arguments": {"city": "NYC", "state": "NY", "unit": "fahrenheit"}}
</tool_call>
<tool_call>
{"name": "get_current_weather", "arguments": {"city": "Baltimore", "state": "MD", "unit": "fahrenheit"}}
</tool_call>
<tool_call>
{"name": "get_current_weather", "arguments": {"city": "Minneapolis", "state": "MN", "unit": "fahrenheit"}}
</tool_call>
<tool_call>
{"name": "get_current_weather", "arguments": {"city": "Los Angeles", "state": "CA", "unit": "fahrenheit"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "NYC",
        "state": "NY",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Baltimore",
        "state": "MD",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Minneapolis",
        "state": "MN",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Los Angeles",
        "state": "CA",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-qwen25-four-parallel-calls", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5416", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city", "state", "unit"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"NYC\", \"state\": \"NY\", \"unit\": \"fahrenheit\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"Baltimore\", \"state\": \"MD\", \"unit\": \"fahrenheit\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"Minneapolis\", \"state\": \"MN\", \"unit\": \"fahrenheit\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"Los Angeles\", \"state\": \"CA\", \"unit\": \"fahrenheit\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 11080, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23054, 34, 497, 330, 2454, 788, 330, 23054, 497, 330, 3843, 788, 330, 69, 47910, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 11080, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 33, 87132, 497, 330, 2454, 788, 330, 6076, 497, 330, 3843, 788, 330, 69, 47910, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 11080, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 6217, 38476, 497, 330, 2454, 788, 330, 55181, 497, 330, 3843, 788, 330, 69, 47910, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 455, 11080, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 29596, 11902, 497, 330, 2454, 788, 330, 5049, 497, 330, 3843, 788, 330, 69, 47910, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "NYC", "state": "NY", "unit": "fahrenheit"}}, {"name": "get_current_weather", "arguments": {"city": "Baltimore", "state": "MD", "unit": "fahrenheit"}}, {"name": "get_current_weather", "arguments": {"city": "Minneapolis", "state": "MN", "unit": "fahrenheit"}}, {"name": "get_current_weather", "arguments": {"city": "Los Angeles", "state": "CA", "unit": "fahrenheit"}}]}, "tags": ["parallel-calls"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/sglang-text-before-call-with-space expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-hermes/sglang-text-before-call-with-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'I will check the weather for you.', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>', got 'I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}'
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`one`	fail	content contains '<tool_call>'
`no_leakage`	`special`	fail	content contains '<tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'
`no_leakage`	`rand:1:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:2:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:3:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:4:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:5:8`	fail	content contains '<tool_call>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/sglang-text-before-call-with-space --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "I will check the weather for you.",
+  "content": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "I will check the weather for you.",
+  "content": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86. Tags: single-call, text-before-call, x-no-newlines.

Raw output

I will check the weather for you. <tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call>

Expected parse

{
  "content": "I will check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/sglang-text-before-call-with-space", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_hermes_detector.py#L86", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}], "raw_output": "I will check the weather for you. <tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}</tool_call>", "output_token_ids": [40, 686, 1779, 279, 9104, 369, 498, 13, 220, 151657, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 52854, 16032, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "I will check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-no-newlines"], "notes": "Content is the text before the call; the separating space is whitespace (soft-v1). output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/single-call-no-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/single-call-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/single-call-no-thinking --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/single-call-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call"], "notes": "enable_thinking=false: the template pre-fills an empty think block in the generation prompt. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/single-call-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/single-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"; tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"; tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "\nThe user wants the current weather in Paris. I'll call get_weather.\n"
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:2:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/single-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "\nThe user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "\nThe user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Strategies: token, rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "\nThe user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, reasoning.

Raw output

<think>
The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/single-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [151667, 198, 785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/string-escapes-code expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/string-escapes-code
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\me\\\\hello.py", "content": "def main():\\n\\tprint(\\"hi \\\\u00e9\\")\\n\\treturn {\\"ok\\": True}\\n"}'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/string-escapes-code --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "def main():\n\tprint(\"hi \\u00e9\")\n\treturn {\"ok\": True}\n",
-        "path": "C:\\Users\\me\\hello.py"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "write_file"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "def main():\n\tprint(\"hi \\u00e9\")\n\treturn {\"ok\": True}\n",
        "path": "C:\\Users\\me\\hello.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, string-escapes.

Raw output

<tool_call>
{"name": "write_file", "arguments": {"path": "C:\\Users\\me\\hello.py", "content": "def main():\n\tprint(\"hi \\u00e9\")\n\treturn {\"ok\": True}\n"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "def main():\n\tprint(\"hi \\u00e9\")\n\treturn {\"ok\": True}\n",
        "path": "C:\\Users\\me\\hello.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/string-escapes-code", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}], "raw_output": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"C:\\\\Users\\\\me\\\\hello.py\", \"content\": \"def main():\\n\\tprint(\\\"hi \\\\u00e9\\\")\\n\\treturn {\\\"ok\\\": True}\\n\"}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 4934, 2458, 497, 330, 16370, 788, 5212, 2343, 788, 330, 34, 23817, 7137, 3422, 2660, 3422, 14990, 7197, 497, 330, 1796, 788, 330, 750, 1887, 4555, 59, 77, 4955, 1350, 36014, 6023, 24984, 84, 15, 15, 68, 24, 62705, 59, 77, 4955, 689, 314, 2105, 562, 11693, 3007, 11035, 77, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "C:\\Users\\me\\hello.py", "content": "def main():\n\tprint(\"hi \\u00e9\")\n\treturn {\"ok\": True}\n"}}]}, "tags": ["single-call", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/text-before-call-no-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/text-before-call-no-thinking
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	content: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`special`	fail	content: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`expected_match`	`rand:2:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`expected_match`	`rand:3:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got 'Let me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:1:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:3:8 (fail): content: one None, got 'Let me check the current conditions in'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '', got '{"city": "London"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call.

Raw output

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/text-before-call-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/text-before-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`one`	fail	content: expected 'Let me check the current conditions in London.', got None; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`special`	fail	content: expected 'Let me check the current conditions in London.', got None; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:2:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:4:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got '\nI need live weather data for London.\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got '\n\nLet me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:1:8 (fail): content: one None, got '\n\nLet me check the current conditions in London'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:2:8 (fail): content: one None, got '\n\nLet me check the current conditions in'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:3:8 (fail): content: one None, got '\n\nLet me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:4:8 (fail): content: one None, got '\n\nLet me check the current conditions'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:5:8 (fail): content: one None, got '\n\nLet me check the current conditions in London'; tool_calls[0].arguments: one '', got '{"city": "London"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/text-before-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": null,
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London.\n",
+  "reasoning_content": "\nI need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, text-before-call, reasoning.

Raw output

<think>
I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/text-before-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nI need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [151667, 198, 40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/thinking2507-multi-turn expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/thinking2507-multi-turn
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'; tool_calls[0].arguments: expected '{"query": "café Zürich"}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'; tool_calls[0].arguments: expected '{"query": "café Zürich"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'It is 20C. Now the café search.', got 'It is 20C. Now the café search.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"query": "café Zürich"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"query": "café Zürich"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"query": "café Zürich"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich"}' \| rand:2:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"query": "café Zürich"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "café Zürich"}'
`arguments_json`	`one`	fail	[0] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] search: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] search: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-multi-turn --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {
       "arguments": {
-        "query": "café Zürich"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "search"

Strategies: nonstream, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {

Strategies: token, rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "It is 20C. Now the café search.",
+  "content": "\n\n",
+  "reasoning_content": "It is 20C. Now the café search.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, multi-turn, reasoning, reasoning-prefilled, unicode.

Raw output

It is 20C. Now the café search.
</think>

<tool_call>
{"name": "search", "arguments": {"query": "café Zürich"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C. Now the café search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-multi-turn", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "It is 20C. Now the café search.\n</think>\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café Zürich\"}}\n</tool_call>", "output_token_ids": [2132, 374, 220, 17, 15, 34, 13, 4695, 279, 51950, 2711, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 1863, 5186, 713, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "It is 20C. Now the café search.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "reasoning", "reasoning-prefilled", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "deepseek-r1",
    "tool_call_parser": "deepseekv3"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198,
      151667,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/thinking2507-parallel-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order

Checks that did not pass for qwen3-hermes/thinking2507-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'; tool_calls: expected ['get_weather', 'search'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,25 +1,11 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: parallel-calls, reasoning, reasoning-prefilled, unicode, nested-json, string-escapes.

Raw output

I should call the tools.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-parallel-calls", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "I should call the tools.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}}\n</tool_call>", "output_token_ids": [40, 1265, 1618, 279, 7375, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858, 7245, 15862, 55853, 330, 24608, 788, 5212, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 3417, 532, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "deepseek-r1",
    "tool_call_parser": "deepseekv3"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198,
      151667,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/thinking2507-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/thinking2507-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"; tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"; tool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "c"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Raw output

The user wants the current weather in Paris. I'll call get_weather.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-single-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "The user wants the current weather in Paris. I'll call get_weather.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris\", \"unit\": \"c\"}}\n</tool_call>", "output_token_ids": [785, 1196, 6801, 279, 1482, 9104, 304, 12095, 13, 358, 3278, 1618, 633, 69364, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604, 497, 330, 3843, 788, 330, 66, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "deepseek-r1",
    "tool_call_parser": "deepseekv3"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198,
      151667,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/thinking2507-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/thinking2507-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`one`	fail	content: expected 'Let me check the current conditions in London.', got None; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`special`	fail	content: expected 'Let me check the current conditions in London.', got None; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:3:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:4:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the current conditions in London.', got '\n\nLet me'; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London.\n'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me check the current conditions in London'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Let me check the current conditions in London.', got '\n\nLet me'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got '\n\nLet me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:1:8 (fail): content: one None, got '\n\nLet me'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:2:8 (fail): content: one None, got '\n\nLet me check the current conditions in London.\n'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:3:8 (fail): content: one None, got '\n\nLet me check'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:4:8 (fail): content: one None, got '\n\nLet me check the current conditions in London'; tool_calls[0].arguments: one '', got '{"city": "London"}' \| rand:5:8 (fail): content: one None, got '\n\nLet me'; tool_calls[0].arguments: one '', got '{"city": "London"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": null,
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: token, rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": "\n\nLet me check the current conditions in London.\n",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json. Tags: single-call, text-before-call, reasoning, reasoning-prefilled.

Raw output

I need live weather data for London.
</think>

Let me check the current conditions in London.
<tool_call>
{"name": "get_weather", "arguments": {"city": "London"}}
</tool_call>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-text-before-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "I need live weather data for London.\n</think>\n\nLet me check the current conditions in London.\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"London\"}}\n</tool_call>", "output_token_ids": [40, 1184, 3887, 9104, 821, 369, 7148, 624, 151668, 271, 10061, 752, 1779, 279, 1482, 4682, 304, 7148, 624, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 39572, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "deepseek-r1",
    "tool_call_parser": "deepseekv3"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198,
      151667,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/truncated-mid-arguments-no-thinking expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-no-thinking
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream [], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris"}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-no-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-no-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Paris", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 59604], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after '\"city\": \"Par' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/truncated-mid-arguments-thinking expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-mid-arguments-thinking
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool-call JSON: no complete call exists.
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream [], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{}' \| rand:1:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: $: 'city' is a required property
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: $: 'city' is a required property
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: $: 'city' is a required property
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: $: 'city' is a required property
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: $: 'city' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-mid-arguments-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:4:8

{
  "content": null,
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8

{
  "content": "\n\n",
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: rand:2:8, rand:3:8, rand:5:8

{
  "content": null,
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": "\nI should call the tool.\n",
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, reasoning.

Expected graceful failure: Output cut by max_tokens inside the tool-call JSON: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<think>
I should call the tool.
</think>

<tool_call>
{"name": "get_weather", "arguments": {"city

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-mid-arguments-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<think>\nI should call the tool.\n</think>\n\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city", "output_token_ids": [151667, 198, 40, 1265, 1618, 279, 5392, 624, 151668, 271, 151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected_error": {"reason": "Output cut by max_tokens inside the tool-call JSON: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning"], "notes": "Token prefix of the full render, cut right after '\"arguments\": {\"ci' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/truncated-second-parallel-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got ''
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got ''
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-hermes/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "query": "café"
+      },
+      "name": "search"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, parallel-calls, unicode.

Raw output

<tool_call>
{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}
</tool_call>
<tool_call>
{"name": "search", "arguments": {"query": "café

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-second-parallel-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Zürich\", \"unit\": \"c\"}}\n</tool_call>\n<tool_call>\n{\"name\": \"search\", \"arguments\": {\"query\": \"café", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 57, 5186, 713, 497, 330, 3843, 788, 330, 66, 95642, 151658, 198, 151657, 198, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 1631, 788, 330, 924, 58858], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut mid-arguments. A correct parser keeps the complete call and drops the unterminated one without leaking its markup (the fix proposed in https://github.com/sgl-project/sglang/issues/30480: valid calls survive a truncated neighbour). Token prefix of the full render, cut right after '\"query\": \"caf' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-hermes/unicode-emoji-thinking expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/unicode-emoji-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'; tool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got ''
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'; tool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got ''
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Translate the mixed-script greeting (日本語, العربية, emoji 🌸).', got '\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got ''
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' \| rand:1:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' \| rand:5:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls[0].arguments: one '', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}'
`arguments_json`	`one`	fail	[0] translate: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] translate: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] translate: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] translate: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/edge.jsonl --id qwen3-hermes/unicode-emoji-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "reasoning_content": "\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {
       "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "translate"

Strategies: nonstream, rand:2:8, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "reasoning_content": "\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {

Strategies: token, rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
+  "content": "\n\n",
+  "reasoning_content": "\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: single-call, unicode, reasoning.

Raw output

<think>
Translate the mixed-script greeting (日本語, العربية, emoji 🌸).
</think>

<tool_call>
{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).",
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/unicode-emoji-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text", "target"]}}}], "raw_output": "<think>\nTranslate the mixed-script greeting (日本語, العربية, emoji 🌸).\n</think>\n\n<tool_call>\n{\"name\": \"translate\", \"arguments\": {\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}}\n</tool_call>", "output_token_ids": [151667, 198, 27473, 279, 9519, 65960, 42113, 320, 101059, 102819, 11, 129071, 11, 42365, 11162, 234, 116, 4292, 151668, 271, 151657, 198, 4913, 606, 788, 330, 14045, 497, 330, 16370, 788, 5212, 1318, 788, 330, 89015, 11162, 234, 116, 23364, 126860, 124671, 1959, 94880, 586, 51950, 61804, 102, 378, 235, 145851, 497, 330, 5657, 788, 330, 450, 95642, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "Translate the mixed-script greeting (日本語, العربية, emoji 🌸).", "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-bug-19056-boolean-argument expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/vllm-bug-19056-boolean-argument
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"trigger": true}', got ''
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"trigger": true}', got ''
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"trigger": true}', got ''
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"trigger": true}', got ''
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}'
`arguments_json`	`one`	fail	[0] final_answer: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] final_answer: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] final_answer: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] final_answer: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-bug-19056-boolean-argument --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "trigger": true
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "final_answer"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "trigger": true
      },
      "name": "final_answer"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L83. Tags: single-call, regression.

Raw output

<tool_call>
{"name": "final_answer", "arguments": {"trigger": true}}
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "trigger": true
      },
      "name": "final_answer"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-bug-19056-boolean-argument", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L83", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "final_answer", "parameters": {"type": "object", "properties": {"trigger": {"type": "boolean"}}, "required": ["trigger"]}}}], "raw_output": "<tool_call>\n{\"name\": \"final_answer\", \"arguments\": {\"trigger\": true}}\n</tool_call>", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 11822, 28534, 497, 330, 16370, 788, 5212, 18468, 788, 830, 11248, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "final_answer", "arguments": {"trigger": true}}]}, "tags": ["single-call", "regression"], "notes": "Regression test for https://github.com/vllm-project/vllm/issues/19056. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-content-and-call-single-chunk expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-hermes/vllm-content-and-call-single-chunk
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>'; tool_calls: expected ['f'], got []
`expected_match`	`one`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`special`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`token`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'Hi!', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'; tool_calls: expected ['f'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>', got 'Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}'
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`one`	fail	content contains '<tool_call>'
`no_leakage`	`special`	fail	content contains '<tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'
`no_leakage`	`rand:1:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:2:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:3:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:4:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:5:8`	fail	content contains '<tool_call>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-content-and-call-single-chunk --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Hi!",
+  "content": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "f"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Hi!",
+  "content": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "f"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L397. Tags: single-call, text-before-call, numeric-arguments, x-no-newlines.

Raw output

Hi!<tool_call>{"name": "f", "arguments": {"x": 1}}</tool_call>

Expected parse

{
  "content": "Hi!",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "f"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-content-and-call-single-chunk", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L397", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "f", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}}}}}], "raw_output": "Hi!<tool_call>{\"name\": \"f\", \"arguments\": {\"x\": 1}}</tool_call>", "output_token_ids": [13048, 0, 151657, 4913, 606, 788, 330, 69, 497, 330, 16370, 788, 5212, 87, 788, 220, 16, 3417, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Hi!", "reasoning_content": null, "tool_calls": [{"name": "f", "arguments": {"x": 1}}]}, "tags": ["single-call", "text-before-call", "numeric-arguments", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-invalid-json-missing-brace expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for qwen3-hermes/vllm-invalid-json-missing-brace
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['final_answer'] for The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream [], got ['final_answer']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream [], got ['final_answer']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}' \| rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"trigger": true}'
`arguments_json`	`one`	fail	[0] final_answer: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] final_answer: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] final_answer: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] final_answer: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/malformed.jsonl --id qwen3-hermes/vllm-invalid-json-missing-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-hermes/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "final_answer"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "trigger": true
      },
      "name": "final_answer"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381. Tags: malformed, truncated.

Expected graceful failure: The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<tool_call>
{"name": "final_answer", "arguments": {"trigger": true}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-invalid-json-missing-brace", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L381", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "final_answer", "parameters": {"type": "object", "properties": {"trigger": {"type": "boolean"}}, "required": ["trigger"]}}}], "raw_output": "<tool_call>\n{\"name\": \"final_answer\", \"arguments\": {\"trigger\": true}", "output_token_ids": [151657, 198, 4913, 606, 788, 330, 11822, 28534, 497, 330, 16370, 788, 5212, 18468, 788, 830, 92], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected_error": {"reason": "The call JSON is missing its closing brace and there is no </tool_call>; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated"], "notes": "vLLM asserts tools_called is false for this output. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-no-newlines-no-spaces expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-hermes/vllm-no-newlines-no-spaces
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`one`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`special`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`token`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'; tool_calls: expected ['get_current_temperature'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>', got '<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}'
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`one`	fail	content contains '<tool_call>'
`no_leakage`	`special`	fail	content contains '<tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'
`no_leakage`	`rand:1:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:2:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:3:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:4:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:5:8`	fail	content contains '<tool_call>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-no-newlines-no-spaces --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "San Francisco, California, United States",
-        "unit": "celsius"
-      },
-      "name": "get_current_temperature"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "San Francisco, California, United States",
-        "unit": "celsius"
-      },
-      "name": "get_current_temperature"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L117. Tags: single-call, x-no-newlines.

Raw output

<tool_call>{"name": "get_current_temperature","arguments": {"location":"San Francisco, California, United States", "unit": "celsius"}}</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "San Francisco, California, United States",
        "unit": "celsius"
      },
      "name": "get_current_temperature"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-no-newlines-no-spaces", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L117", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_current_temperature", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}, "required": ["location"]}}}], "raw_output": "<tool_call>{\"name\": \"get_current_temperature\",\"arguments\": {\"location\":\"San Francisco, California, United States\", \"unit\": \"celsius\"}}</tool_call>", "output_token_ids": [151657, 4913, 606, 788, 330, 455, 11080, 53525, 2198, 16370, 788, 5212, 2527, 3252, 23729, 12879, 11, 7043, 11, 3639, 4180, 497, 330, 3843, 788, 330, 66, 40247, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_temperature", "arguments": {"location": "San Francisco, California, United States", "unit": "celsius"}}]}, "tags": ["single-call", "x-no-newlines"], "notes": "No newline inside the tags and irregular JSON spacing; the tags alone delimit the call. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-text-then-call-no-separator expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-hermes/vllm-text-then-call-no-separator
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'Sure, let me check the weather.', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>', got 'Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}'
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`one`	fail	content contains '<tool_call>'
`no_leakage`	`special`	fail	content contains '<tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'
`no_leakage`	`rand:1:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:2:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:3:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:4:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:5:8`	fail	content contains '<tool_call>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-text-then-call-no-separator --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Sure, let me check the weather.",
+  "content": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "NYC"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Sure, let me check the weather.",
+  "content": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "NYC"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L227. Tags: single-call, text-before-call, x-no-newlines.

Raw output

Sure, let me check the weather.<tool_call>{"name": "get_weather", "arguments": {"city": "NYC"}}</tool_call>

Expected parse

{
  "content": "Sure, let me check the weather.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "NYC"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-text-then-call-no-separator", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L227", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Sure, let me check the weather.<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}</tool_call>", "output_token_ids": [39814, 11, 1077, 752, 1779, 279, 9104, 13, 151657, 4913, 606, 788, 330, 455, 69364, 497, 330, 16370, 788, 5212, 8926, 788, 330, 23054, 34, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": "Sure, let me check the weather.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "NYC"}}]}, "tags": ["single-call", "text-before-call", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-hermes/vllm-two-calls-no-separator expected_match, stream_equals_nonstream, no_leakage, parallel_order

Checks that did not pass for qwen3-hermes/vllm-two-calls-no-separator
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`one`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`special`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`token`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'; tool_calls: expected ['search', 'search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>', got '<tool_call>{"name": "search", "arguments": {"q": "cats"}}<tool_call>{"name": "search", "arguments": {"q": "dogs"}}'
`no_leakage`	`nonstream`	fail	content contains '<tool_call>'; content contains '</tool_call>'
`no_leakage`	`one`	fail	content contains '<tool_call>'
`no_leakage`	`special`	fail	content contains '<tool_call>'
`no_leakage`	`token`	fail	content contains '<tool_call>'
`no_leakage`	`rand:1:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:2:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:3:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:4:8`	fail	content contains '<tool_call>'
`no_leakage`	`rand:5:8`	fail	content contains '<tool_call>'
`parallel_order`	`nonstream`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`one`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`special`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`rand:1:8`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`rand:2:8`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`rand:3:8`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`rand:4:8`	fail	expected 2 calls ['search', 'search'], got 0 []
`parallel_order`	`rand:5:8`	fail	expected 2 calls ['search', 'search'], got 0 []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/engine-tests.jsonl --id qwen3-hermes/vllm-two-calls-no-separator --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,18 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}</tool_call><tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "q": "cats"
-      },
-      "name": "search"
-    },
-    {
-      "arguments": {
-        "q": "dogs"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,18 +1,5 @@
 {
-  "content": null,
+  "content": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "q": "cats"
-      },
-      "name": "search"
-    },
-    {
-      "arguments": {
-        "q": "dogs"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<tool_call>{"name": "search", "arguments": {"q": "cats"}}</tool_call><tool_call>{"name": "search", "arguments": {"q": "dogs"}}</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "q": "cats"
      },
      "name": "search"
    },
    {
      "arguments": {
        "q": "dogs"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/vllm-two-calls-no-separator", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_hermes_tool_parser.py#L258", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"q": {"type": "string"}}, "required": ["q"]}}}], "raw_output": "<tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"cats\"}}</tool_call><tool_call>{\"name\": \"search\", \"arguments\": {\"q\": \"dogs\"}}</tool_call>", "output_token_ids": [151657, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 80, 788, 330, 37452, 30975, 151658, 151657, 4913, 606, 788, 330, 1836, 497, 330, 16370, 788, 5212, 80, 788, 330, 80034, 30975, 151658], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"q": "cats"}}, {"name": "search", "arguments": {"q": "dogs"}}]}, "tags": ["parallel-calls", "x-no-newlines"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      151667,
      271,
      151668,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-hermes/multi-turn-final-answer-thinking expected_match

Checks that did not pass for qwen3-hermes/multi-turn-final-answer-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'It is 20 °C in Zürich right now.', got '\n\nIt is 20 °C in Zürich right now.'; reasoning_content (whitespace only): expected 'The tool returned 20C.', got '\nThe tool returned 20C.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/multiturn.jsonl --id qwen3-hermes/multi-turn-final-answer-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-hermes/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "It is 20 °C in Zürich right now.",
-  "reasoning_content": "The tool returned 20C.",
+  "content": "\n\nIt is 20 °C in Zürich right now.",
+  "reasoning_content": "\nThe tool returned 20C.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, multi-turn, reasoning, unicode.

Raw output

<think>
The tool returned 20C.
</think>

It is 20 °C in Zürich right now.

Expected parse

{
  "content": "It is 20 °C in Zürich right now.",
  "reasoning_content": "The tool returned 20C.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/multi-turn-final-answer-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe tool returned 20C.\n</think>\n\nIt is 20 °C in Zürich right now.", "output_token_ids": [151667, 198, 785, 5392, 5927, 220, 17, 15, 34, 624, 151668, 271, 2132, 374, 220, 17, 15, 36286, 34, 304, 1863, 5186, 713, 1290, 1431, 13], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "It is 20 °C in Zürich right now.", "reasoning_content": "The tool returned 20C.", "tool_calls": []}, "tags": ["no-call", "multi-turn", "reasoning", "unicode"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-hermes/no-call-thinking expected_match

Checks that did not pass for qwen3-hermes/no-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got '\nThe user is just greeting me. No tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/basic.jsonl --id qwen3-hermes/no-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-hermes/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Hello! How can I help you today?",
-  "reasoning_content": "The user is just greeting me. No tool is needed.",
+  "content": "\n\nHello! How can I help you today?",
+  "reasoning_content": "\nThe user is just greeting me. No tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: no-call, reasoning.

Raw output

<think>
The user is just greeting me. No tool is needed.
</think>

Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/no-call-thinking", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user is just greeting me. No tool is needed.\n</think>\n\nHello! How can I help you today?", "output_token_ids": [151667, 198, 785, 1196, 374, 1101, 42113, 752, 13, 2308, 5392, 374, 4362, 624, 151668, 271, 9707, 0, 2585, 646, 358, 1492, 498, 3351, 30], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "The user is just greeting me. No tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-hermes/thinking2507-no-call expected_match

Checks that did not pass for qwen3-hermes/thinking2507-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected '2 + 2 = 4.', got '\n\n2 + 2 = 4.'; reasoning_content (whitespace only): expected 'Simple arithmetic, no tool needed.', got 'Simple arithmetic, no tool needed.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/thinking-2507.jsonl --id qwen3-hermes/thinking2507-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-hermes/thinking-2507.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "2 + 2 = 4.",
-  "reasoning_content": "Simple arithmetic, no tool needed.",
+  "content": "\n\n2 + 2 = 4.",
+  "reasoning_content": "Simple arithmetic, no tool needed.\n",
   "tool_calls": []
 }

Fixture

Raw output

Simple arithmetic, no tool needed.
</think>

2 + 2 = 4.

Expected parse

{
  "content": "2 + 2 = 4.",
  "reasoning_content": "Simple arithmetic, no tool needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/thinking2507-no-call", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-4B-Thinking-2507", "Qwen/Qwen3-30B-A3B-Thinking-2507", "Qwen/Qwen3-235B-A22B-Thinking-2507"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507/blob/768f209d9ea81521153ed38c47d515654e938aea/tokenizer_config.json", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Simple arithmetic, no tool needed.\n</think>\n\n2 + 2 = 4.", "output_token_ids": [16374, 34784, 11, 902, 5392, 4362, 624, 151668, 271, 17, 488, 220, 17, 284, 220, 19, 13], "tokenizer": {"repo": "Qwen/Qwen3-4B-Thinking-2507", "revision": "768f209d9ea81521153ed38c47d515654e938aea", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "2 + 2 = 4.", "reasoning_content": "Simple arithmetic, no tool needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "deepseek-r1",
    "tool_call_parser": "deepseekv3"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3802169b2a02b81e6adb7ab4f64f91ff02db753c8c3a64a01c35192d3a61d8d7",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198,
      151667,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151643,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-4B-Thinking-2507",
    "revision": "768f209d9ea81521153ed38c47d515654e938aea",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-hermes/truncated-in-reasoning expected_match

Checks that did not pass for qwen3-hermes/truncated-in-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather', got '\nThe user wants the current weather'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-hermes/truncated.jsonl --id qwen3-hermes/truncated-in-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-hermes/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather",
+  "reasoning_content": "\nThe user wants the current weather",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json. Tags: truncated, reasoning.

Raw output

<think>
The user wants the current weather

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-hermes/truncated-in-reasoning", "family": "qwen3-hermes", "models": ["Qwen/Qwen3-0.6B", "Qwen/Qwen3-8B", "Qwen/Qwen3-32B", "Qwen/Qwen3-30B-A3B", "Qwen/Qwen3-235B-A22B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/c1899de289a04d12100db370d81485cdf75e47ca/tokenizer_config.json", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-hermes/build.py", "template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<think>\nThe user wants the current weather", "output_token_ids": [151667, 198, 785, 1196, 6801, 279, 1482, 9104], "tokenizer": {"repo": "Qwen/Qwen3-0.6B", "revision": "c1899de289a04d12100db370d81485cdf75e47ca", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather", "tool_calls": []}, "tags": ["truncated", "reasoning"], "notes": "Cut inside <think> (no </think> yet): the partial text is reasoning, not content. Token prefix of the full render, cut right after 'current weather' appears. History render of the official template; Qwen3 generates this exact layout."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "model_type": "qwen3"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-0.6B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-0.6B",
    "revision": "c1899de289a04d12100db370d81485cdf75e47ca",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen25Detector",
  "tool_call_parser": "qwen25",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model