Qwen XML tool calls (Qwen3-Coder, Qwen3.5/3.6/3.8) on sglang 0.5.20

fail 35% strict pass 19 pass · 27 soft pass · 8 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	19	27	7	36%
`expected_error`	0	0	1	0%
`stream_equals_nonstream`	40	6	8	74%
`split_invariance`	54	0	0	100%
`no_leakage`	50	0	4	93%
`arguments_json`	47	0	2	96%
`arguments_schema`	46	0	3	94%
`parallel_order`	6	0	1	86%

Fixtures needing attention

fail qwen3-xml/bug-coder-missing-tool-call-opener expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-xml/bug-coder-missing-tool-call-opener
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para...; tool_calls: expected ['glob'], got []
`expected_match`	`one`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`special`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`token`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para..., got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n'; tool_calls: nonstream [], got ['glob']
`no_leakage`	`nonstream`	fail	content contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/bug-coder-missing-tool-call-opener --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 18 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.",
+  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<parameter=pattern>\n**/*.md\n</parameter>\n</function>\n</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "pattern": "**/*.md"
-      },
-      "name": "glob"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.",
+  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18530. Tags: single-call, text-before-call, malformed, regression, x-bare-function.

Raw output

I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.

<function=glob>
<parameter=pattern>
**/*.md
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "pattern": "**/*.md"
      },
      "name": "glob"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/bug-coder-missing-tool-call-opener", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18530", "revision": "issue-18530", "license": "NOASSERTION", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py"}, "tools": [{"type": "function", "function": {"name": "glob", "parameters": {"type": "object", "properties": {"pattern": {"type": "string"}}, "required": ["pattern"]}, "description": "Find files by glob pattern"}}], "raw_output": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<parameter=pattern>\n**/*.md\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 686, 1156, 1779, 369, 678, 73192, 3542, 304, 279, 1482, 6220, 323, 1181, 1186, 75701, 11, 323, 1221, 1140, 1105, 1667, 279, 13206, 5392, 382, 27, 1688, 37162, 1684, 397, 27, 16181, 28, 14339, 397, 334, 23540, 2277, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.", "reasoning_content": null, "tool_calls": [{"name": "glob", "arguments": {"pattern": "**/*.md"}}]}, "tags": ["single-call", "text-before-call", "malformed", "regression", "x-bare-function"], "notes": "A real qwen3-coder:30b generation captured by the reporter with raw: true (temperature 0, seed 1): the model omitted the <tool_call> opener after a prose preamble."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-xml/bug-coder-text-after-call expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/bug-coder-text-after-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'beforeafter', got 'before'
`stream_equals_nonstream`	`one`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'before', got 'beforeafter'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'before', got 'beforeafter'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/bug-coder-text-after-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 19 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "beforeafter",
+  "content": "before",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "beforeafter",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/40739. Tags: single-call, text-before-call, text-after-call, regression, x-no-newlines.

Raw output

before<tool_call><function=get_weather><parameter=city>Paris</parameter></function></tool_call>after

Expected parse

{
  "content": "beforeafter",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/bug-coder-text-after-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/40739", "revision": "issue-40739", "license": "NOASSERTION", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}], "raw_output": "before<tool_call><function=get_weather><parameter=city>Paris</parameter></function></tool_call>after", "output_token_ids": [14801, 151657, 27, 1688, 28280, 69364, 1784, 16181, 28, 8926, 29, 59604, 522, 16181, 1472, 1688, 29, 151658, 10694], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "beforeafter", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "text-before-call", "text-after-call", "regression", "x-no-newlines"], "notes": "The issue's input string (no newlines inside the call). Its expected behaviour: streaming and non-streaming both return content 'beforeafter' and the call."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-xml/coder-marker-in-arguments expected_match, stream_equals_nonstream, no_leakage, arguments_schema

Checks that did not pass for qwen3-xml/coder-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"path": "x.md", "content": "Close with </function> then </tool_call>; open with <function=x>."}', got '{"path": "x.md", "content": "Close with "}'
`expected_match`	`one`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`special`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`token`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`rand:1:8`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`rand:2:8`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`rand:3:8`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`rand:4:8`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`expected_match`	`rand:5:8`	fail	content: expected None, got '; open with .\n</parameter>\n\n'; tool_calls: expected ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '; open with .\n</parameter>\n\n'; tool_calls: nonstream ['write_file'], got ['write_file', 'x']
`no_leakage`	`one`	fail	content contains '</parameter>'
`no_leakage`	`special`	fail	content contains '</parameter>'
`no_leakage`	`token`	fail	content contains '</parameter>'
`no_leakage`	`rand:1:8`	fail	content contains '</parameter>'
`no_leakage`	`rand:2:8`	fail	content contains '</parameter>'
`no_leakage`	`rand:3:8`	fail	content contains '</parameter>'
`no_leakage`	`rand:4:8`	fail	content contains '</parameter>'
`no_leakage`	`rand:5:8`	fail	content contains '</parameter>'
`arguments_schema`	`one`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`special`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`token`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`rand:1:8`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`rand:2:8`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`rand:3:8`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`rand:4:8`	fail	[1] 'x' is not an offered tool
`arguments_schema`	`rand:5:8`	fail	[1] 'x' is not an offered tool

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/coder-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 32 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Close with </function> then </tool_call>; open with <function=x>.",
+        "content": "Close with ",
         "path": "x.md"
       },

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,16 @@
 {
-  "content": null,
+  "content": "; open with .\n</parameter>\n\n",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "content": "Close with </function> then </tool_call>; open with <function=x>.",
+        "content": "Close with ",
         "path": "x.md"
       },
       "name": "write_file"
+    },
+    {
+      "arguments": {},
+      "name": "x"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<tool_call>
<function=write_file>
<parameter=path>
x.md
</parameter>
<parameter=content>
Close with </function> then </tool_call>; open with <function=x>.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "Close with </function> then </tool_call>; open with <function=x>.",
        "path": "x.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/coder-marker-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "<tool_call>\n<function=write_file>\n<parameter=path>\nx.md\n</parameter>\n<parameter=content>\nClose with </function> then </tool_call>; open with <function=x>.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [151657, 198, 27, 1688, 28, 4934, 2458, 397, 27, 16181, 81940, 397, 87, 21324, 198, 522, 16181, 397, 27, 16181, 28, 1796, 397, 7925, 448, 690, 1688, 29, 1221, 220, 151658, 26, 1787, 448, 366, 1688, 25387, 29816, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "x.md", "content": "Close with </function> then </tool_call>; open with <function=x>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3-Coder-30B-A3B-Instruct chat template. XML-looking markers inside a value, not at line starts. '</tool_call>' is an added token, so it appears as its marker id in output_token_ids."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

fail qwen3-xml/llamacpp-coder-bare-function expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for qwen3-xml/llamacpp-coder-bare-function
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>'; tool_calls: expected ['special_function'], got []
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>', got '\n\n\n'; tool_calls: nonstream [], got ['special_function']
`no_leakage`	`nonstream`	fail	content contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-coder-bare-function --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3559-L3569. Tags: single-call, malformed, numeric-arguments, x-bare-function.

Raw output

<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-coder-bare-function", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3559-L3569", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [27, 1688, 14149, 2964, 9174, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "malformed", "numeric-arguments", "x-bare-function"], "notes": "'Some models skip the opening <tool_call> and go straight to <function=>' (also Ollama #17353)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail qwen3-xml/llamacpp-coder-bare-function-parallel expected_match, stream_equals_nonstream, no_leakage, parallel_order

Checks that did not pass for qwen3-xml/llamacpp-coder-bare-function-parallel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n'; tool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function_with_opt']
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n', got '\n\n\n\n'; tool_calls: nonstream ['special_function_with_opt'], got ['special_function', 'special_function_with_opt']
`no_leakage`	`nonstream`	fail	content contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'
`parallel_order`	`nonstream`	fail	expected 2 calls ['special_function', 'special_function_with_opt'], got 1 ['special_function_with_opt']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-coder-bare-function-parallel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": null,
+  "content": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n",
   "reasoning_content": null,
   "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
     {
       "arguments": {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3586-L3612. Tags: parallel-calls, malformed, numeric-arguments, x-bare-function.

Raw output

<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>
<tool_call>
<function=special_function_with_opt>
<parameter=arg1>
1
</parameter>
<parameter=arg2>
2
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-coder-bare-function-parallel", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3586-L3612", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=special_function_with_opt>\n<parameter=arg1>\n1\n</parameter>\n<parameter=arg2>\n2\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [27, 1688, 14149, 2964, 9174, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 522, 1688, 397, 151658, 198, 151657, 198, 27, 1688, 14149, 2964, 9174, 6615, 15032, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 27, 16181, 28, 858, 17, 397, 17, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "malformed", "numeric-arguments", "x-bare-function"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail qwen3-xml/q38-marker-in-arguments expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for qwen3-xml/q38-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}'
`expected_match`	`one`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`special`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`token`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`rand:1:8`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`rand:2:8`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`rand:3:8`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`rand:4:8`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`expected_match`	`rand:5:8`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the note.', got 'Write the note.\n'; tool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"path": "notes.md", "content": "Close tags inline: "}', got '{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "notes.md", "content": "Close tags inline: "}, "path": " are text here."}': Extra data: line 1 column 55 (char 54))
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
-  "reasoning_content": "Write the note.",
+  "content": "\n\n",
+  "reasoning_content": "Write the note.\n",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here.",
+        "content": "Close tags inline: ",
         "path": "notes.md"
       },

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
-  "reasoning_content": "Write the note.",
+  "content": "\n\n",
+  "reasoning_content": "Write the note.\n",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here.",
-        "path": "notes.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"notes.md\", \"content\": \"Close tags inline: \"}, \"path\": \" are text here.\"}"
       },
       "name": "write_file"

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: single-call, reasoning, reasoning-prefilled, marker-in-arguments.

Raw output

Write the note.
</think>

<tool_call>
<function=write_file>
<parameter=path>
notes.md
</parameter>
<parameter=content>
Close tags inline: </parameter> and </function> and <parameter=path> are text here.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the note.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here.",
        "path": "notes.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-marker-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write the note.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nnotes.md\n</parameter>\n<parameter=content>\nClose tags inline: </parameter> and </function> and <parameter=path> are text here.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7734, 279, 5020, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 17742, 20668, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 7675, 9212, 7147, 25, 672, 15704, 29, 321, 672, 1628, 29, 321, 361, 15704, 79114, 29, 513, 1414, 1532, 13, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Write the note.", "tool_calls": [{"name": "write_file", "arguments": {"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. The value contains XML-looking markers that are not at a line start. The structural delimiter is '\\n</parameter>\\n', so a parser that splits on a bare '</parameter>' cuts the value short."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

fail qwen3-xml/q38-tool-call-token-in-arguments expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/q38-tool-call-token-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'; tool_calls[0].arguments: expected '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Document the format.', got 'Document the format.\n'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... "}', got '{"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-tool-call-token-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
-  "reasoning_content": "Document the format.",
+  "content": "\n\n",
+  "reasoning_content": "Document the format.\n",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks.",
+        "content": "Qwen wraps calls in <tool_call> ... ",
         "path": "fmt.md"
       },

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Document the format.",
+  "content": "\n\n",
+  "reasoning_content": "Document the format.\n",
   "tool_calls": [
     {

Fixture

Raw output

Document the format.
</think>

<tool_call>
<function=write_file>
<parameter=path>
fmt.md
</parameter>
<parameter=content>
Qwen wraps calls in <tool_call> ... </tool_call> blocks.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Document the format.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks.",
        "path": "fmt.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-tool-call-token-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Document the format.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nfmt.md\n</parameter>\n<parameter=content>\nQwen wraps calls in <tool_call> ... </tool_call> blocks.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7288, 279, 3443, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 12143, 20668, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 48, 16451, 38394, 6526, 303, 220, 248058, 2423, 220, 248059, 9714, 13, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Document the format.", "tool_calls": [{"name": "write_file", "arguments": {"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. '<tool_call>' and '</tool_call>' are added tokens, so the tokenizer maps them to their marker ids even inside the argument value (see output_token_ids). Only the XML structure tells them apart."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

fail qwen3-xml/q38-truncated-in-parameter expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for qwen3-xml/q38-truncated-in-parameter
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`one`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`special`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`token`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['write_file'] for Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"path": "/tmp/build.sh", "content": "rm -rf /"}', got '{"path": "/tmp/build.sh"'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "/tmp/build.sh"': Expecting ',' delimiter: line 1 column 25 (char 24))
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-truncated-in-parameter --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 22 of fixtures/qwen3-xml/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": "\n\n",
  "reasoning_content": "Write the script.\n",
  "tool_calls": [
    {
      "arguments": {
        "content": "rm -rf /",
        "path": "/tmp/build.sh"
      },
      "name": "write_file"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "\n\n",
  "reasoning_content": "Write the script.\n",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"path\": \"/tmp/build.sh\""
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: truncated, reasoning, reasoning-prefilled.

Expected graceful failure: Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699). (accept: no_tool_calls, content_passthrough, exception).

Raw output

Write the script.
</think>

<tool_call>
<function=write_file>
<parameter=path>
/tmp/build.sh
</parameter>
<parameter=content>
rm -rf /

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-truncated-in-parameter", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write the script.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\n/tmp/build.sh\n</parameter>\n<parameter=content>\nrm -rf /", "output_token_ids": [7734, 279, 5148, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 55101, 29594, 2318, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 8454, 471, 7799, 593], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected_error": {"reason": "Output stopped (max_tokens) inside a parameter value, before </parameter> and </function>. The call is incomplete; a truncated 'rm -rf /tmp/build' must not become a complete 'rm -rf /' (vLLM #57699).", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "reasoning-prefilled"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/coder-next-parallel-calls expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/coder-next-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/coder-next-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 34 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Lyon"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Nice"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/a7fbcb5c0e12d62a448eaa0e260346bf5dcc0feb/chat_template.jinja. Tags: parallel-calls.

Raw output

<tool_call>
<function=get_weather>
<parameter=city>
Lyon
</parameter>
</function>
</tool_call>
<tool_call>
<function=get_weather>
<parameter=city>
Nice
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Lyon"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Nice"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/coder-next-parallel-calls", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-Next"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-Coder-Next/blob/a7fbcb5c0e12d62a448eaa0e260346bf5dcc0feb/chat_template.jinja", "revision": "a7fbcb5c0e12d62a448eaa0e260346bf5dcc0feb", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c79a833039a43602150cce0902403d6e376c50930c1b2a139b2964e1f0c322a0"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "<tool_call>\n<function=get_weather>\n<parameter=city>\nLyon\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=get_weather>\n<parameter=city>\nNice\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [151657, 198, 27, 1688, 28280, 69364, 397, 27, 16181, 28, 8926, 397, 43, 24990, 198, 522, 16181, 397, 522, 1688, 397, 151658, 198, 151657, 198, 27, 1688, 28280, 69364, 397, 27, 16181, 28, 8926, 397, 44978, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-Next", "revision": "a7fbcb5c0e12d62a448eaa0e260346bf5dcc0feb", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Lyon"}}, {"name": "get_weather", "arguments": {"city": "Nice"}}]}, "tags": ["parallel-calls"], "notes": "History render of the official Qwen/Qwen3-Coder-Next chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c79a833039a43602150cce0902403d6e376c50930c1b2a139b2964e1f0c322a0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3NextForCausalLM"
    ],
    "model_type": "qwen3_next"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-Next",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-Next",
    "revision": "a7fbcb5c0e12d62a448eaa0e260346bf5dcc0feb",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/coder-parallel-calls expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/coder-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/coder-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 27 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja. Tags: parallel-calls, unicode, nested-json, string-escapes.

Raw output

<tool_call>
<function=get_weather>
<parameter=city>
Zürich
</parameter>
<parameter=unit>
c
</parameter>
</function>
</tool_call>
<tool_call>
<function=search>
<parameter=query>
café "best"
</parameter>
<parameter=filters>
{"tags": ["a", "b"], "max": 3}
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/coder-parallel-calls", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "<tool_call>\n<function=get_weather>\n<parameter=city>\nZürich\n</parameter>\n<parameter=unit>\nc\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=search>\n<parameter=query>\ncafé \"best\"\n</parameter>\n<parameter=filters>\n{\"tags\": [\"a\", \"b\"], \"max\": 3}\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [151657, 198, 27, 1688, 28280, 69364, 397, 27, 16181, 28, 8926, 397, 57, 5186, 713, 198, 522, 16181, 397, 27, 16181, 28, 3843, 397, 66, 198, 522, 16181, 397, 522, 1688, 397, 151658, 198, 151657, 198, 27, 1688, 96598, 397, 27, 16181, 79194, 397, 924, 58858, 330, 15862, 698, 522, 16181, 397, 27, 16181, 18111, 8612, 397, 4913, 14082, 788, 4383, 64, 497, 330, 65, 7914, 330, 2810, 788, 220, 18, 532, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official Qwen/Qwen3-Coder-30B-A3B-Instruct chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/coder-text-before-call expected_match

Checks that did not pass for qwen3-xml/coder-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`one`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected "I'll create the file now.", got "I'll create the file now.\n\n"

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/coder-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 28 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll create the file now.",
+  "content": "I'll create the file now.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja. Tags: single-call, text-before-call, x-multiline-string.

Raw output

I'll create the file now.

<tool_call>
<function=write_file>
<parameter=path>
hello.py
</parameter>
<parameter=content>
def greet(name: str) -> str:
    """Say hi."""
    return f"Hello, {name}!"


print(greet("Zoë"))
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "I'll create the file now.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "def greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))",
        "path": "hello.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/coder-text-before-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "I'll create the file now.\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nhello.py\n</parameter>\n<parameter=content>\ndef greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 3278, 1855, 279, 1034, 1431, 382, 151657, 198, 27, 1688, 28, 4934, 2458, 397, 27, 16181, 81940, 397, 14990, 7197, 198, 522, 16181, 397, 27, 16181, 28, 1796, 397, 750, 40786, 3153, 25, 607, 8, 1464, 607, 510, 262, 4210, 45764, 15588, 14442, 262, 470, 282, 1, 9707, 11, 314, 606, 92, 8958, 1406, 1350, 3268, 3744, 445, 57, 78, 12179, 5455, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "I'll create the file now.", "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "hello.py", "content": "def greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))"}}]}, "tags": ["single-call", "text-before-call", "x-multiline-string"], "notes": "History render of the official Qwen/Qwen3-Coder-30B-A3B-Instruct chat template. The Qwen3-Coder template writes '\\n' + content + '\\n' and then '\\n<tool_call>', so the content is followed by '\\n\\n' of markup."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/llamacpp-q35-call-inside-think expected_match

Checks that did not pass for qwen3-xml/llamacpp-q35-call-inside-think
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'Need to inspect the current directory.', got 'Need to inspect the current directory.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-q35-call-inside-think --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Need to inspect the current directory.",
+  "reasoning_content": "Need to inspect the current directory.\n",
   "tool_calls": [
     {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2323-L2340. Tags: single-call, reasoning, reasoning-prefilled, malformed, x-call-inside-think.

Raw output

Need to inspect the current directory.
<tool_call>
<function=run_in_terminal>
<parameter=command>
pwd
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Need to inspect the current directory.",
  "tool_calls": [
    {
      "arguments": {
        "command": "pwd"
      },
      "name": "run_in_terminal"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-q35-call-inside-think", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2323-L2340", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "run_in_terminal", "parameters": {"type": "object", "properties": {"command": {"type": "string", "description": "Shell command to run"}}, "required": ["command"]}, "description": "Run a shell command."}}], "raw_output": "Need to inspect the current directory.\n<tool_call>\n<function=run_in_terminal>\n<parameter=command>\npwd\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [22916, 310, 23331, 279, 1428, 6025, 13, 198, 248058, 198, 27, 1628, 28, 5917, 1201, 62138, 29, 198, 27, 15704, 28, 5454, 29, 198, 24751, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Need to inspect the current directory.", "tool_calls": [{"name": "run_in_terminal", "arguments": {"command": "pwd"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "malformed", "x-call-inside-think"], "notes": "The model starts a tool call without closing the prefilled <think> block. llama.cpp's test ('a tool call ends the prefilled thinking block, with or without a closing </think>') treats <tool_call> as ending the reasoning. This is the most-reported failure for the family: https://github.com/ggml-org/llama.cpp/issues/20837, https://github.com/vllm-project/vllm/issues/39056."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-xml/llamacpp-q35-parallel-no-think expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/llamacpp-q35-parallel-no-think
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-q35-parallel-no-think --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2176-L2198. Tags: parallel-calls, numeric-arguments.

Raw output

<tool_call>
<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>
<tool_call>
<function=special_function_with_opt>
<parameter=arg1>
1
</parameter>
<parameter=arg2>
2
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-q35-parallel-no-think", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2176-L2198", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "<tool_call>\n<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=special_function_with_opt>\n<parameter=arg1>\n1\n</parameter>\n<parameter=arg2>\n2\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [248058, 198, 27, 1628, 13744, 2869, 8901, 29, 198, 27, 15704, 28, 832, 16, 29, 198, 16, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 198, 248058, 198, 27, 1628, 13744, 2869, 8901, 6406, 14596, 29, 198, 27, 15704, 28, 832, 16, 29, 198, 16, 198, 510, 15704, 29, 198, 27, 15704, 28, 832, 17, 29, 198, 17, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "numeric-arguments"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      248068,
      271,
      248069,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

soft pass qwen3-xml/llamacpp-q35-reasoning-content-call expected_match

Checks that did not pass for qwen3-xml/llamacpp-q35-reasoning-content-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me inspect it now.', got '\n\nLet me inspect it now.\n'; reasoning_content (whitespace only): expected 'I should inspect the directory.', got 'I should inspect the directory.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-q35-reasoning-content-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me inspect it now.",
-  "reasoning_content": "I should inspect the directory.",
+  "content": "\n\nLet me inspect it now.\n",
+  "reasoning_content": "I should inspect the directory.\n",
   "tool_calls": [
     {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2459-L2480. Tags: single-call, reasoning, reasoning-prefilled, text-before-call.

Raw output

I should inspect the directory.
</think>

Let me inspect it now.
<tool_call>
<function=run_in_terminal>
<parameter=command>
pwd
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "Let me inspect it now.",
  "reasoning_content": "I should inspect the directory.",
  "tool_calls": [
    {
      "arguments": {
        "command": "pwd"
      },
      "name": "run_in_terminal"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-q35-reasoning-content-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2459-L2480", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "run_in_terminal", "parameters": {"type": "object", "properties": {"command": {"type": "string", "description": "Shell command to run"}}, "required": ["command"]}, "description": "Run a shell command."}}], "raw_output": "I should inspect the directory.\n</think>\n\nLet me inspect it now.\n<tool_call>\n<function=run_in_terminal>\n<parameter=command>\npwd\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 1220, 23331, 279, 6025, 13, 198, 248069, 271, 9764, 728, 23331, 424, 1381, 13, 198, 248058, 198, 27, 1628, 28, 5917, 1201, 62138, 29, 198, 27, 15704, 28, 5454, 29, 198, 24751, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Let me inspect it now.", "reasoning_content": "I should inspect the directory.", "tool_calls": [{"name": "run_in_terminal", "arguments": {"command": "pwd"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "text-before-call"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-xml/llamacpp-q35-reasoning-then-call expected_match

Checks that did not pass for qwen3-xml/llamacpp-q35-reasoning-then-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected "I'm\nthinking", got "I'm\nthinking\n"

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-q35-reasoning-then-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I'm\nthinking",
+  "content": "\n\n",
+  "reasoning_content": "I'm\nthinking\n",
   "tool_calls": [
     {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2164-L2174. Tags: single-call, reasoning, reasoning-prefilled, numeric-arguments.

Raw output

I'm
thinking
</think>

<tool_call>
<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I'm\nthinking",
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/llamacpp-q35-reasoning-then-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2164-L2174", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "I'm\nthinking\n</think>\n\n<tool_call>\n<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 2688, 198, 79420, 198, 248069, 271, 248058, 198, 27, 1628, 13744, 2869, 8901, 29, 198, 27, 15704, 28, 832, 16, 29, 198, 16, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "I'm\nthinking", "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "numeric-arguments"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass qwen3-xml/q35-python-literals expected_match

Checks that did not pass for qwen3-xml/q35-python-literals
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set a repeating alarm.', got 'Set a repeating alarm.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q35-python-literals --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 24 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Set a repeating alarm.",
+  "content": "\n\n",
+  "reasoning_content": "Set a repeating alarm.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.5-9B/blob/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja. Tags: single-call, reasoning, reasoning-prefilled, numeric-arguments, x-python-literals.

Raw output

Set a repeating alarm.
</think>

<tool_call>
<function=set_alarm>
<parameter=hour>
6
</parameter>
<parameter=minute>
0
</parameter>
<parameter=repeat>
True
</parameter>
<parameter=label>
gym
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Set a repeating alarm.",
  "tool_calls": [
    {
      "arguments": {
        "hour": 6,
        "label": "gym",
        "minute": 0,
        "repeat": true
      },
      "name": "set_alarm"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q35-python-literals", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.5-9B/blob/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Set a repeating alarm.\n</think>\n\n<tool_call>\n<function=set_alarm>\n<parameter=hour>\n6\n</parameter>\n<parameter=minute>\n0\n</parameter>\n<parameter=repeat>\nTrue\n</parameter>\n<parameter=label>\ngym\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [1591, 264, 38495, 16136, 13, 198, 248069, 271, 248058, 198, 27, 1628, 66473, 64049, 29, 198, 27, 15704, 28, 23899, 29, 198, 21, 198, 510, 15704, 29, 198, 27, 15704, 28, 41143, 29, 198, 15, 198, 510, 15704, 29, 198, 27, 15704, 28, 29625, 29, 198, 2434, 198, 510, 15704, 29, 198, 27, 15704, 95277, 29, 198, 70, 1544, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Set a repeating alarm.", "tool_calls": [{"name": "set_alarm", "arguments": {"hour": 6, "minute": 0, "repeat": true, "label": "gym"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "numeric-arguments", "x-python-literals"], "notes": "History render of the official Qwen/Qwen3.5-9B chat template. The Qwen3.5-9B template writes Python literals ('True'); with a boolean schema the value is true."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q35-single-call expected_match

Checks that did not pass for qwen3-xml/q35-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Weather lookup.', got 'Weather lookup.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q35-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 25 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Weather lookup.",
+  "content": "\n\n",
+  "reasoning_content": "Weather lookup.\n",
   "tool_calls": [
     {

Fixture

Raw output

Weather lookup.
</think>

<tool_call>
<function=get_weather>
<parameter=city>
Zürich
</parameter>
<parameter=unit>
c
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Weather lookup.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q35-single-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.5-9B/blob/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Weather lookup.\n</think>\n\n<tool_call>\n<function=get_weather>\n<parameter=city>\nZürich\n</parameter>\n<parameter=unit>\nc\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [28034, 18054, 13, 198, 248069, 271, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 57, 5021, 693, 198, 510, 15704, 29, 198, 27, 15704, 28, 3715, 29, 198, 66, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Weather lookup.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "unicode"], "notes": "History render of the official Qwen/Qwen3.5-9B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.5-9B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.5-9B",
    "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-empty-arguments expected_match

Checks that did not pass for qwen3-xml/q38-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Get the time.', got 'Get the time.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Get the time.",
+  "content": "\n\n",
+  "reasoning_content": "Get the time.\n",
   "tool_calls": [
     {

Fixture

Raw output

Get the time.
</think>

<tool_call>
<function=get_time>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Get the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-empty-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Get the time.\n</think>\n\n<tool_call>\n<function=get_time>\n</function>\n</tool_call>", "output_token_ids": [1882, 279, 854, 13, 198, 248069, 271, 248058, 198, 27, 1628, 27362, 2914, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Get the time.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "empty-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-empty-reasoning expected_match

Checks that did not pass for qwen3-xml/q38-empty-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-empty-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": null,
+  "content": "\n\n",
+  "reasoning_content": "\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: single-call, empty-arguments, reasoning-prefilled.

Raw output

</think>

<tool_call>
<function=get_time>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-empty-reasoning", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "\n</think>\n\n<tool_call>\n<function=get_time>\n</function>\n</tool_call>", "output_token_ids": [198, 248069, 271, 248058, 198, 27, 1628, 27362, 2914, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning-prefilled"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. The prompt/completion boundary fell inside a merged BPE token, so the rendered text was sliced after the generation prompt and the remainder re-encoded. Thinking enabled but the reasoning is empty: the completion starts with '\\n</think>'."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-long-arguments expected_match

Checks that did not pass for qwen3-xml/q38-long-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the module.', got 'Write the module.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-long-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Write the module.",
+  "content": "\n\n",
+  "reasoning_content": "Write the module.\n",
   "tool_calls": [
     {

Fixture

Raw output

Write the module.
</think>

<tool_call>
<function=write_file>
<parameter=path>
steps.py
</parameter>
<parameter=content>
"""Generated module used as a long tool argument."""

from __future__ import annotations

def step_00(x: int) -> int:
    return x + 0

def step_01(x: int) -> int:
    return x + 1

def step_02(x: int) -> int:
    return x + 2

def step_03(x: int) -> int:
    return x + 3

def step_04(x: int) -> int:
    return x + 4

def step_05(x: int) -> int:
    return x + 5

def step_06(x: int) -> int:
    return x + 6

def step_07(x: int) -> int:
    return x + 7

def step_08(x: int) -> int:
    return x + 8

def step_09(x: int) -> int:
    return x + 9

def step_10(x: int) -> int:
    return x + 10

def step_11(x: int) -> int:
    return x + 11

def step_12(x: int) -> int:
    return x + 12

def step_13(x: int) -> int:
    return x + 13

def step_14(x: int) -> int:
    return x + 14

def step_15(x: int) -> int:
    return x + 15

def step_16(x: int) -> int:
    return x + 16

def step_17(x: int) -> int:
    return x + 17

def step_18(x: int) -> int:
    return x + 18

def step_19(x: int) -> int:
    return x + 19

def step_20(x: int) -> int:
    return x + 20

def step_21(x: int) -> int:
    return x + 21

def step_22(x: int) -> int:
    return x + 22

def step_23(x: int) -> int:
    return x + 23

def step_24(x: int) -> int:
    return x + 24

def step_25(x: int) -> int:
    return x + 25

def step_26(x: int) -> int:
    return x + 26

def step_27(x: int) -> int:
    return x + 27

def step_28(x: int) -> int:
    return x + 28

def step_29(x: int) -> int:
    return x + 29

def step_30(x: int) -> int:
    return x + 30

def step_31(x: int) -> int:
    return x + 31

def step_32(x: int) -> int:
    return x + 32

def step_33(x: int) -> int:
    return x + 33

def step_34(x: int) -> int:
    return x + 34

def step_35(x: int) -> int:
    return x + 35

def step_36(x: int) -> int:
    return x + 36

def step_37(x: int) -> int:
    return x + 37

def step_38(x: int) -> int:
    return x + 38

def step_39(x: int) -> int:
    return x + 39

def step_40(x: int) -> int:
    return x + 40

def step_41(x: int) -> int:
    return x + 41

def step_42(x: int) -> int:
    return x + 42

def step_43(x: int) -> int:
    return x + 43

def step_44(x: int) -> int:
    return x + 44

def step_45(x: int) -> int:
    return x + 45

def step_46(x: int) -> int:
    return x + 46

def step_47(x: int) -> int:
    return x + 47

def step_48(x: int) -> int:
    return x + 48

def step_49(x: int) -> int:
    return x + 49

def step_50(x: int) -> int:
    return x + 50

def step_51(x: int) -> int:
    return x + 51

def step_52(x: int) -> int:
    return x + 52

def step_53(x: int) -> int:
    return x + 53

def step_54(x: int) -> int:
    return x + 54

def step_55(x: int) -> int:
    return x + 55

def step_56(x: int) -> int:
    return x + 56

def step_57(x: int) -> int:
    return x + 57

def step_58(x: int) -> int:
    return x + 58

def step_59(x: int) -> int:
    return x + 59

def step_60(x: int) -> int:
    return x + 60

def step_61(x: int) -> int:
    return x + 61

def step_62(x: int) -> int:
    return x + 62

def step_63(x: int) -> int:
    return x + 63

def step_64(x: int) -> int:
    return x + 64

def step_65(x: int) -> int:
    return x + 65

def step_66(x: int) -> int:
    return x + 66

def step_67(x: int) -> int:
    return x + 67

def step_68(x: int) -> int:
    return x + 68

def step_69(x: int) -> int:
    return x + 69

def step_70(x: int) -> int:
    return x + 70

def step_71(x: int) -> int:
    return x + 71

def step_72(x: int) -> int:
    return x + 72

def step_73(x: int) -> int:
    return x + 73

def step_74(x: int) -> int:
    return x + 74

def step_75(x: int) -> int:
    return x + 75

def step_76(x: int) -> int:
    return x + 76

def step_77(x: int) -> int:
    return x + 77

def step_78(x: int) -> int:
    return x + 78

def step_79(x: int) -> int:
    return x + 79

</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the module.",
  "tool_calls": [
    {
      "arguments": {
        "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    return x + 0\n\ndef step_01(x: int) -> int:\n    return x + 1\n\ndef step_02(x: int) -> int:\n    return x + 2\n\ndef step_03(x: int) -> int:\n    return x + 3\n\ndef step_04(x: int) -> int:\n    return x + 4\n\ndef step_05(x: int) -> int:\n    return x + 5\n\ndef step_06(x: int) -> int:\n    return x + 6\n\ndef step_07(x: int) -> int:\n    return x + 7\n\ndef step_08(x: int) -> int:\n    return x + 8\n\ndef step_09(x: int) -> int:\n    return x + 9\n\ndef step_10(x: int) -> int:\n    return x + 10\n\ndef step_11(x: int) -> int:\n    return x + 11\n\ndef step_12(x: int) -> int:\n    return x + 12\n\ndef step_13(x: int) -> int:\n    return x + 13\n\ndef step_14(x: int) -> int:\n    return x + 14\n\ndef step_15(x: int) -> int:\n    return x + 15\n\ndef step_16(x: int) -> int:\n    return x + 16\n\ndef step_17(x: int) -> int:\n    return x + 17\n\ndef step_18(x: int) -> int:\n    return x + 18\n\ndef step_19(x: int) -> int:\n    return x + 19\n\ndef step_20(x: int) -> int:\n    return x + 20\n\ndef step_21(x: int) -> int:\n    return x + 21\n\ndef step_22(x: int) -> int:\n    return x + 22\n\ndef step_23(x: int) -> int:\n    return x + 23\n\ndef step_24(x: int) -> int:\n    return x + 24\n\ndef step_25(x: int) -> int:\n    return x + 25\n\ndef step_26(x: int) -> int:\n    return x + 26\n\ndef step_27(x: int) -> int:\n    return x + 27\n\ndef step_28(x: int) -> int:\n    return x + 28\n\ndef step_29(x: int) -> int:\n    return x + 29\n\ndef step_30(x: int) -> int:\n    return x + 30\n\ndef step_31(x: int) -> int:\n    return x + 31\n\ndef step_32(x: int) -> int:\n    return x + 32\n\ndef step_33(x: int) -> int:\n    return x + 33\n\ndef step_34(x: int) -> int:\n    return x + 34\n\ndef step_35(x: int) -> int:\n    return x + 35\n\ndef step_36(x: int) -> int:\n    return x + 36\n\ndef step_37(x: int) -> int:\n    return x + 37\n\ndef step_38(x: int) -> int:\n    return x + 38\n\ndef step_39(x: int) -> int:\n    return x + 39\n\ndef step_40(x: int) -> int:\n    return x + 40\n\ndef step_41(x: int) -> int:\n    return x + 41\n\ndef step_42(x: int) -> int:\n    return x + 42\n\ndef step_43(x: int) -> int:\n    return x + 43\n\ndef step_44(x: int) -> int:\n    return x + 44\n\ndef step_45(x: int) -> int:\n    return x + 45\n\ndef step_46(x: int) -> int:\n    return x + 46\n\ndef step_47(x: int) -> int:\n    return x + 47\n\ndef step_48(x: int) -> int:\n    return x + 48\n\ndef step_49(x: int) -> int:\n    return x + 49\n\ndef step_50(x: int) -> int:\n    return x + 50\n\ndef step_51(x: int) -> int:\n    return x + 51\n\ndef step_52(x: int) -> int:\n    return x + 52\n\ndef step_53(x: int) -> int:\n    return x + 53\n\ndef step_54(x: int) -> int:\n    return x + 54\n\ndef step_55(x: int) -> int:\n    return x + 55\n\ndef step_56(x: int) -> int:\n    return x + 56\n\ndef step_57(x: int) -> int:\n    return x + 57\n\ndef step_58(x: int) -> int:\n    return x + 58\n\ndef step_59(x: int) -> int:\n    return x + 59\n\ndef step_60(x: int) -> int:\n    return x + 60\n\ndef step_61(x: int) -> int:\n    return x + 61\n\ndef step_62(x: int) -> int:\n    return x + 62\n\ndef step_63(x: int) -> int:\n    return x + 63\n\ndef step_64(x: int) -> int:\n    return x + 64\n\ndef step_65(x: int) -> int:\n    return x + 65\n\ndef step_66(x: int) -> int:\n    return x + 66\n\ndef step_67(x: int) -> int:\n    return x + 67\n\ndef step_68(x: int) -> int:\n    return x + 68\n\ndef step_69(x: int) -> int:\n    return x + 69\n\ndef step_70(x: int) -> int:\n    return x + 70\n\ndef step_71(x: int) -> int:\n    return x + 71\n\ndef step_72(x: int) -> int:\n    return x + 72\n\ndef step_73(x: int) -> int:\n    return x + 73\n\ndef step_74(x: int) -> int:\n    return x + 74\n\ndef step_75(x: int) -> int:\n    return x + 75\n\ndef step_76(x: int) -> int:\n    return x + 76\n\ndef step_77(x: int) -> int:\n    return x + 77\n\ndef step_78(x: int) -> int:\n    return x + 78\n\ndef step_79(x: int) -> int:\n    return x + 79\n",
        "path": "steps.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-long-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write the module.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nsteps.py\n</parameter>\n<parameter=content>\n\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    return x + 0\n\ndef step_01(x: int) -> int:\n    return x + 1\n\ndef step_02(x: int) -> int:\n    return x + 2\n\ndef step_03(x: int) -> int:\n    return x + 3\n\ndef step_04(x: int) -> int:\n    return x + 4\n\ndef step_05(x: int) -> int:\n    return x + 5\n\ndef step_06(x: int) -> int:\n    return x + 6\n\ndef step_07(x: int) -> int:\n    return x + 7\n\ndef step_08(x: int) -> int:\n    return x + 8\n\ndef step_09(x: int) -> int:\n    return x + 9\n\ndef step_10(x: int) -> int:\n    return x + 10\n\ndef step_11(x: int) -> int:\n    return x + 11\n\ndef step_12(x: int) -> int:\n    return x + 12\n\ndef step_13(x: int) -> int:\n    return x + 13\n\ndef step_14(x: int) -> int:\n    return x + 14\n\ndef step_15(x: int) -> int:\n    return x + 15\n\ndef step_16(x: int) -> int:\n    return x + 16\n\ndef step_17(x: int) -> int:\n    return x + 17\n\ndef step_18(x: int) -> int:\n    return x + 18\n\ndef step_19(x: int) -> int:\n    return x + 19\n\ndef step_20(x: int) -> int:\n    return x + 20\n\ndef step_21(x: int) -> int:\n    return x + 21\n\ndef step_22(x: int) -> int:\n    return x + 22\n\ndef step_23(x: int) -> int:\n    return x + 23\n\ndef step_24(x: int) -> int:\n    return x + 24\n\ndef step_25(x: int) -> int:\n    return x + 25\n\ndef step_26(x: int) -> int:\n    return x + 26\n\ndef step_27(x: int) -> int:\n    return x + 27\n\ndef step_28(x: int) -> int:\n    return x + 28\n\ndef step_29(x: int) -> int:\n    return x + 29\n\ndef step_30(x: int) -> int:\n    return x + 30\n\ndef step_31(x: int) -> int:\n    return x + 31\n\ndef step_32(x: int) -> int:\n    return x + 32\n\ndef step_33(x: int) -> int:\n    return x + 33\n\ndef step_34(x: int) -> int:\n    return x + 34\n\ndef step_35(x: int) -> int:\n    return x + 35\n\ndef step_36(x: int) -> int:\n    return x + 36\n\ndef step_37(x: int) -> int:\n    return x + 37\n\ndef step_38(x: int) -> int:\n    return x + 38\n\ndef step_39(x: int) -> int:\n    return x + 39\n\ndef step_40(x: int) -> int:\n    return x + 40\n\ndef step_41(x: int) -> int:\n    return x + 41\n\ndef step_42(x: int) -> int:\n    return x + 42\n\ndef step_43(x: int) -> int:\n    return x + 43\n\ndef step_44(x: int) -> int:\n    return x + 44\n\ndef step_45(x: int) -> int:\n    return x + 45\n\ndef step_46(x: int) -> int:\n    return x + 46\n\ndef step_47(x: int) -> int:\n    return x + 47\n\ndef step_48(x: int) -> int:\n    return x + 48\n\ndef step_49(x: int) -> int:\n    return x + 49\n\ndef step_50(x: int) -> int:\n    return x + 50\n\ndef step_51(x: int) -> int:\n    return x + 51\n\ndef step_52(x: int) -> int:\n    return x + 52\n\ndef step_53(x: int) -> int:\n    return x + 53\n\ndef step_54(x: int) -> int:\n    return x + 54\n\ndef step_55(x: int) -> int:\n    return x + 55\n\ndef step_56(x: int) -> int:\n    return x + 56\n\ndef step_57(x: int) -> int:\n    return x + 57\n\ndef step_58(x: int) -> int:\n    return x + 58\n\ndef step_59(x: int) -> int:\n    return x + 59\n\ndef step_60(x: int) -> int:\n    return x + 60\n\ndef step_61(x: int) -> int:\n    return x + 61\n\ndef step_62(x: int) -> int:\n    return x + 62\n\ndef step_63(x: int) -> int:\n    return x + 63\n\ndef step_64(x: int) -> int:\n    return x + 64\n\ndef step_65(x: int) -> int:\n    return x + 65\n\ndef step_66(x: int) -> int:\n    return x + 66\n\ndef step_67(x: int) -> int:\n    return x + 67\n\ndef step_68(x: int) -> int:\n    return x + 68\n\ndef step_69(x: int) -> int:\n    return x + 69\n\ndef step_70(x: int) -> int:\n    return x + 70\n\ndef step_71(x: int) -> int:\n    return x + 71\n\ndef step_72(x: int) -> int:\n    return x + 72\n\ndef step_73(x: int) -> int:\n    return x + 73\n\ndef step_74(x: int) -> int:\n    return x + 74\n\ndef step_75(x: int) -> int:\n    return x + 75\n\ndef step_76(x: int) -> int:\n    return x + 76\n\ndef step_77(x: int) -> int:\n    return x + 77\n\ndef step_78(x: int) -> int:\n    return x + 78\n\ndef step_79(x: int) -> int:\n    return x + 79\n\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7734, 279, 4536, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 23326, 6971, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 12237, 15280, 4536, 1429, 430, 264, 1248, 5224, 5515, 71483, 271, 1445, 1259, 20409, 548, 1120, 31144, 271, 727, 2923, 62, 15, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 15, 271, 727, 2923, 62, 15, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 271, 727, 2923, 62, 15, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 271, 727, 2923, 62, 15, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 271, 727, 2923, 62, 15, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 271, 727, 2923, 62, 15, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 271, 727, 2923, 62, 15, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 271, 727, 2923, 62, 15, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 271, 727, 2923, 62, 15, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 23, 271, 727, 2923, 62, 15, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 24, 271, 727, 2923, 62, 16, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 15, 271, 727, 2923, 62, 16, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 16, 271, 727, 2923, 62, 16, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 17, 271, 727, 2923, 62, 16, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 18, 271, 727, 2923, 62, 16, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 19, 271, 727, 2923, 62, 16, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 20, 271, 727, 2923, 62, 16, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 21, 271, 727, 2923, 62, 16, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 22, 271, 727, 2923, 62, 16, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 23, 271, 727, 2923, 62, 16, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 16, 24, 271, 727, 2923, 62, 17, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 15, 271, 727, 2923, 62, 17, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 16, 271, 727, 2923, 62, 17, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 17, 271, 727, 2923, 62, 17, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 18, 271, 727, 2923, 62, 17, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 19, 271, 727, 2923, 62, 17, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 20, 271, 727, 2923, 62, 17, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 21, 271, 727, 2923, 62, 17, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 22, 271, 727, 2923, 62, 17, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 23, 271, 727, 2923, 62, 17, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 17, 24, 271, 727, 2923, 62, 18, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 15, 271, 727, 2923, 62, 18, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 16, 271, 727, 2923, 62, 18, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 17, 271, 727, 2923, 62, 18, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 18, 271, 727, 2923, 62, 18, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 19, 271, 727, 2923, 62, 18, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 20, 271, 727, 2923, 62, 18, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 21, 271, 727, 2923, 62, 18, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 22, 271, 727, 2923, 62, 18, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 23, 271, 727, 2923, 62, 18, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 18, 24, 271, 727, 2923, 62, 19, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 15, 271, 727, 2923, 62, 19, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 16, 271, 727, 2923, 62, 19, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 17, 271, 727, 2923, 62, 19, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 18, 271, 727, 2923, 62, 19, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 19, 271, 727, 2923, 62, 19, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 20, 271, 727, 2923, 62, 19, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 21, 271, 727, 2923, 62, 19, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 22, 271, 727, 2923, 62, 19, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 23, 271, 727, 2923, 62, 19, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 19, 24, 271, 727, 2923, 62, 20, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 15, 271, 727, 2923, 62, 20, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 16, 271, 727, 2923, 62, 20, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 17, 271, 727, 2923, 62, 20, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 18, 271, 727, 2923, 62, 20, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 19, 271, 727, 2923, 62, 20, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 20, 271, 727, 2923, 62, 20, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 21, 271, 727, 2923, 62, 20, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 22, 271, 727, 2923, 62, 20, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 23, 271, 727, 2923, 62, 20, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 20, 24, 271, 727, 2923, 62, 21, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 15, 271, 727, 2923, 62, 21, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 16, 271, 727, 2923, 62, 21, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 17, 271, 727, 2923, 62, 21, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 18, 271, 727, 2923, 62, 21, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 19, 271, 727, 2923, 62, 21, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 20, 271, 727, 2923, 62, 21, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 21, 271, 727, 2923, 62, 21, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 22, 271, 727, 2923, 62, 21, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 23, 271, 727, 2923, 62, 21, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 21, 24, 271, 727, 2923, 62, 22, 15, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 15, 271, 727, 2923, 62, 22, 16, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 16, 271, 727, 2923, 62, 22, 17, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 17, 271, 727, 2923, 62, 22, 18, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 18, 271, 727, 2923, 62, 22, 19, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 19, 271, 727, 2923, 62, 22, 20, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 20, 271, 727, 2923, 62, 22, 21, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 21, 271, 727, 2923, 62, 22, 22, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 22, 271, 727, 2923, 62, 22, 23, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 23, 271, 727, 2923, 62, 22, 24, 2007, 25, 514, 8, 1411, 514, 25, 198, 262, 460, 830, 478, 220, 22, 24, 271, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Write the module.", "tool_calls": [{"name": "write_file", "arguments": {"path": "steps.py", "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    return x + 0\n\ndef step_01(x: int) -> int:\n    return x + 1\n\ndef step_02(x: int) -> int:\n    return x + 2\n\ndef step_03(x: int) -> int:\n    return x + 3\n\ndef step_04(x: int) -> int:\n    return x + 4\n\ndef step_05(x: int) -> int:\n    return x + 5\n\ndef step_06(x: int) -> int:\n    return x + 6\n\ndef step_07(x: int) -> int:\n    return x + 7\n\ndef step_08(x: int) -> int:\n    return x + 8\n\ndef step_09(x: int) -> int:\n    return x + 9\n\ndef step_10(x: int) -> int:\n    return x + 10\n\ndef step_11(x: int) -> int:\n    return x + 11\n\ndef step_12(x: int) -> int:\n    return x + 12\n\ndef step_13(x: int) -> int:\n    return x + 13\n\ndef step_14(x: int) -> int:\n    return x + 14\n\ndef step_15(x: int) -> int:\n    return x + 15\n\ndef step_16(x: int) -> int:\n    return x + 16\n\ndef step_17(x: int) -> int:\n    return x + 17\n\ndef step_18(x: int) -> int:\n    return x + 18\n\ndef step_19(x: int) -> int:\n    return x + 19\n\ndef step_20(x: int) -> int:\n    return x + 20\n\ndef step_21(x: int) -> int:\n    return x + 21\n\ndef step_22(x: int) -> int:\n    return x + 22\n\ndef step_23(x: int) -> int:\n    return x + 23\n\ndef step_24(x: int) -> int:\n    return x + 24\n\ndef step_25(x: int) -> int:\n    return x + 25\n\ndef step_26(x: int) -> int:\n    return x + 26\n\ndef step_27(x: int) -> int:\n    return x + 27\n\ndef step_28(x: int) -> int:\n    return x + 28\n\ndef step_29(x: int) -> int:\n    return x + 29\n\ndef step_30(x: int) -> int:\n    return x + 30\n\ndef step_31(x: int) -> int:\n    return x + 31\n\ndef step_32(x: int) -> int:\n    return x + 32\n\ndef step_33(x: int) -> int:\n    return x + 33\n\ndef step_34(x: int) -> int:\n    return x + 34\n\ndef step_35(x: int) -> int:\n    return x + 35\n\ndef step_36(x: int) -> int:\n    return x + 36\n\ndef step_37(x: int) -> int:\n    return x + 37\n\ndef step_38(x: int) -> int:\n    return x + 38\n\ndef step_39(x: int) -> int:\n    return x + 39\n\ndef step_40(x: int) -> int:\n    return x + 40\n\ndef step_41(x: int) -> int:\n    return x + 41\n\ndef step_42(x: int) -> int:\n    return x + 42\n\ndef step_43(x: int) -> int:\n    return x + 43\n\ndef step_44(x: int) -> int:\n    return x + 44\n\ndef step_45(x: int) -> int:\n    return x + 45\n\ndef step_46(x: int) -> int:\n    return x + 46\n\ndef step_47(x: int) -> int:\n    return x + 47\n\ndef step_48(x: int) -> int:\n    return x + 48\n\ndef step_49(x: int) -> int:\n    return x + 49\n\ndef step_50(x: int) -> int:\n    return x + 50\n\ndef step_51(x: int) -> int:\n    return x + 51\n\ndef step_52(x: int) -> int:\n    return x + 52\n\ndef step_53(x: int) -> int:\n    return x + 53\n\ndef step_54(x: int) -> int:\n    return x + 54\n\ndef step_55(x: int) -> int:\n    return x + 55\n\ndef step_56(x: int) -> int:\n    return x + 56\n\ndef step_57(x: int) -> int:\n    return x + 57\n\ndef step_58(x: int) -> int:\n    return x + 58\n\ndef step_59(x: int) -> int:\n    return x + 59\n\ndef step_60(x: int) -> int:\n    return x + 60\n\ndef step_61(x: int) -> int:\n    return x + 61\n\ndef step_62(x: int) -> int:\n    return x + 62\n\ndef step_63(x: int) -> int:\n    return x + 63\n\ndef step_64(x: int) -> int:\n    return x + 64\n\ndef step_65(x: int) -> int:\n    return x + 65\n\ndef step_66(x: int) -> int:\n    return x + 66\n\ndef step_67(x: int) -> int:\n    return x + 67\n\ndef step_68(x: int) -> int:\n    return x + 68\n\ndef step_69(x: int) -> int:\n    return x + 69\n\ndef step_70(x: int) -> int:\n    return x + 70\n\ndef step_71(x: int) -> int:\n    return x + 71\n\ndef step_72(x: int) -> int:\n    return x + 72\n\ndef step_73(x: int) -> int:\n    return x + 73\n\ndef step_74(x: int) -> int:\n    return x + 74\n\ndef step_75(x: int) -> int:\n    return x + 75\n\ndef step_76(x: int) -> int:\n    return x + 76\n\ndef step_77(x: int) -> int:\n    return x + 77\n\ndef step_78(x: int) -> int:\n    return x + 78\n\ndef step_79(x: int) -> int:\n    return x + 79\n"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "long-arguments", "x-multiline-string"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-multiline-string expected_match

Checks that did not pass for qwen3-xml/q38-multiline-string
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write the script.', got 'Write the script.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-multiline-string --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Write the script.",
+  "content": "\n\n",
+  "reasoning_content": "Write the script.\n",
   "tool_calls": [
     {

Fixture

Raw output

Write the script.
</think>

<tool_call>
<function=write_file>
<parameter=path>
greet.py
</parameter>
<parameter=content>
def greet(name: str) -> str:
    """Say hi."""
    return f"Hello, {name}!"


print(greet("Zoë"))
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the script.",
  "tool_calls": [
    {
      "arguments": {
        "content": "def greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))",
        "path": "greet.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-multiline-string", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write the script.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\ngreet.py\n</parameter>\n<parameter=content>\ndef greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7734, 279, 5148, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 70, 3617, 6971, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 727, 39440, 3052, 25, 592, 8, 1411, 592, 25, 198, 262, 4071, 44240, 15131, 71483, 198, 262, 460, 281, 1, 9419, 11, 313, 591, 92, 8687, 1358, 1302, 3163, 3617, 437, 230123, 11831, 2673, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Write the script.", "tool_calls": [{"name": "write_file", "arguments": {"path": "greet.py", "content": "def greet(name: str) -> str:\n    \"\"\"Say hi.\"\"\"\n    return f\"Hello, {name}!\"\n\n\nprint(greet(\"Zoë\"))"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "string-escapes", "x-multiline-string"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-nested-json expected_match

Checks that did not pass for qwen3-xml/q38-nested-json
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Search with structured filters.', got 'Search with structured filters.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-nested-json --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Search with structured filters.",
+  "content": "\n\n",
+  "reasoning_content": "Search with structured filters.\n",
   "tool_calls": [
     {

Fixture

Raw output

Search with structured filters.
</think>

<tool_call>
<function=search>
<parameter=query>
hotels
</parameter>
<parameter=filters>
{"price": {"min": 50, "max": 120.5}, "tags": ["pool", "wifi"], "open": null}
</parameter>
<parameter=sites>
["a.example", "b.example"]
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Search with structured filters.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "open": null,
          "price": {
            "max": 120.5,
            "min": 50
          },
          "tags": [
            "pool",
            "wifi"
          ]
        },
        "query": "hotels",
        "sites": [
          "a.example",
          "b.example"
        ]
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-nested-json", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Search with structured filters.\n</think>\n\n<tool_call>\n<function=search>\n<parameter=query>\nhotels\n</parameter>\n<parameter=filters>\n{\"price\": {\"min\": 50, \"max\": 120.5}, \"tags\": [\"pool\", \"wifi\"], \"open\": null}\n</parameter>\n<parameter=sites>\n[\"a.example\", \"b.example\"]\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [5708, 440, 31838, 13019, 13, 198, 248069, 271, 248058, 198, 27, 1628, 93260, 29, 198, 27, 15704, 76478, 29, 198, 10307, 1943, 198, 510, 15704, 29, 198, 27, 15704, 17572, 8349, 29, 198, 4754, 6352, 763, 5046, 1030, 763, 220, 20, 15, 11, 328, 2721, 763, 220, 16, 17, 15, 13, 20, 2069, 328, 13679, 763, 4241, 9979, 487, 328, 65011, 7664, 328, 2428, 763, 819, 92, 198, 510, 15704, 29, 198, 27, 15704, 13744, 3490, 29, 198, 1143, 64, 7479, 487, 328, 65, 7479, 1293, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Search with structured filters.", "tool_calls": [{"name": "search", "arguments": {"query": "hotels", "filters": {"price": {"min": 50, "max": 120.5}, "tags": ["pool", "wifi"], "open": null}, "sites": ["a.example", "b.example"]}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "nested-json"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Object and array values are rendered with tojson; the parser must decode them as JSON."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-no-call expected_match

Checks that did not pass for qwen3-xml/q38-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Hello! How can I help you today?', got '\n\nHello! How can I help you today?'; reasoning_content (whitespace only): expected 'A simple greeting; no tool is needed.', got 'A simple greeting; no tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Hello! How can I help you today?",
-  "reasoning_content": "A simple greeting; no tool is needed.",
+  "content": "\n\nHello! How can I help you today?",
+  "reasoning_content": "A simple greeting; no tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: no-call, reasoning, reasoning-prefilled.

Raw output

A simple greeting; no tool is needed.
</think>

Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "A simple greeting; no tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-no-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "A simple greeting; no tool is needed.\n</think>\n\nHello! How can I help you today?", "output_token_ids": [32, 4145, 40719, 26, 874, 5224, 369, 4221, 13, 198, 248069, 271, 9419, 0, 2500, 628, 353, 1438, 488, 3242, 30], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "A simple greeting; no tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-numeric-and-literal-arguments expected_match

Checks that did not pass for qwen3-xml/q38-numeric-and-literal-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Set the alarm.', got 'Set the alarm.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-numeric-and-literal-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 13 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Set the alarm.",
+  "content": "\n\n",
+  "reasoning_content": "Set the alarm.\n",
   "tool_calls": [
     {

Fixture

Raw output

Set the alarm.
</think>

<tool_call>
<function=set_alarm>
<parameter=hour>
7
</parameter>
<parameter=minute>
30
</parameter>
<parameter=volume>
0.75
</parameter>
<parameter=repeat>
true
</parameter>
<parameter=label>
null
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Set the alarm.",
  "tool_calls": [
    {
      "arguments": {
        "hour": 7,
        "label": null,
        "minute": 30,
        "repeat": true,
        "volume": 0.75
      },
      "name": "set_alarm"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-numeric-and-literal-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Set the alarm.\n</think>\n\n<tool_call>\n<function=set_alarm>\n<parameter=hour>\n7\n</parameter>\n<parameter=minute>\n30\n</parameter>\n<parameter=volume>\n0.75\n</parameter>\n<parameter=repeat>\ntrue\n</parameter>\n<parameter=label>\nnull\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [1591, 279, 16136, 13, 198, 248069, 271, 248058, 198, 27, 1628, 66473, 64049, 29, 198, 27, 15704, 28, 23899, 29, 198, 22, 198, 510, 15704, 29, 198, 27, 15704, 28, 41143, 29, 198, 18, 15, 198, 510, 15704, 29, 198, 27, 15704, 28, 24266, 29, 198, 15, 13, 22, 20, 198, 510, 15704, 29, 198, 27, 15704, 28, 29625, 29, 198, 1802, 198, 510, 15704, 29, 198, 27, 15704, 95277, 29, 198, 2827, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Set the alarm.", "tool_calls": [{"name": "set_alarm", "arguments": {"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label": null}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "numeric-arguments", "x-schema-coercion"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Scalars are raw text; integer/number/boolean/null come back only through the tool's JSON schema."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-numeric-looking-strings expected_match

Checks that did not pass for qwen3-xml/q38-numeric-looking-strings
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Look it up.', got 'Look it up.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-numeric-looking-strings --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Look it up.",
+  "content": "\n\n",
+  "reasoning_content": "Look it up.\n",
   "tool_calls": [
     {

Fixture

Raw output

Look it up.
</think>

<tool_call>
<function=lookup>
<parameter=zip>
02139
</parameter>
<parameter=account_id>
123456789012345678901
</parameter>
<parameter=flag>
true
</parameter>
<parameter=payload>
{"a": 1}
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Look it up.",
  "tool_calls": [
    {
      "arguments": {
        "account_id": "123456789012345678901",
        "flag": "true",
        "payload": "{\"a\": 1}",
        "zip": "02139"
      },
      "name": "lookup"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-numeric-looking-strings", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Look it up.\n</think>\n\n<tool_call>\n<function=lookup>\n<parameter=zip>\n02139\n</parameter>\n<parameter=account_id>\n123456789012345678901\n</parameter>\n<parameter=flag>\ntrue\n</parameter>\n<parameter=payload>\n{\"a\": 1}\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [10071, 424, 685, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 20377, 29, 198, 27, 15704, 28, 9669, 29, 198, 15, 17, 16, 18, 24, 198, 510, 15704, 29, 198, 27, 15704, 28, 4459, 816, 29, 198, 16, 17, 18, 19, 20, 21, 22, 23, 24, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 15, 16, 198, 510, 15704, 29, 198, 27, 15704, 28, 9610, 29, 198, 1802, 198, 510, 15704, 29, 198, 27, 15704, 16874, 6771, 29, 198, 4754, 64, 763, 220, 16, 92, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Look it up.", "tool_calls": [{"name": "lookup", "arguments": {"zip": "02139", "account_id": "123456789012345678901", "flag": "true", "payload": "{\"a\": 1}"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "x-schema-coercion"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Every parameter is typed 'string' in the schema, so '02139', 'true' and '{\"a\": 1}' must stay strings."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-parallel-calls expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/q38-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I should call the tools.",
+  "content": "\n\n\n",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: parallel-calls, reasoning, reasoning-prefilled, unicode, nested-json, string-escapes.

Raw output

I should call the tools.
</think>

<tool_call>
<function=get_weather>
<parameter=city>
Zürich
</parameter>
<parameter=unit>
c
</parameter>
</function>
</tool_call>
<tool_call>
<function=search>
<parameter=query>
café "best"
</parameter>
<parameter=filters>
{"tags": ["a", "b"], "max": 3}
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-parallel-calls", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "I should call the tools.\n</think>\n\n<tool_call>\n<function=get_weather>\n<parameter=city>\nZürich\n</parameter>\n<parameter=unit>\nc\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=search>\n<parameter=query>\ncafé \"best\"\n</parameter>\n<parameter=filters>\n{\"tags\": [\"a\", \"b\"], \"max\": 3}\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 1220, 1562, 279, 7141, 13, 198, 248069, 271, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 57, 5021, 693, 198, 510, 15704, 29, 198, 27, 15704, 28, 3715, 29, 198, 66, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 198, 248058, 198, 27, 1628, 93260, 29, 198, 27, 15704, 76478, 29, 198, 895, 56868, 328, 15395, 1, 198, 510, 15704, 29, 198, 27, 15704, 17572, 8349, 29, 198, 4754, 13679, 763, 4241, 64, 487, 328, 65, 7664, 328, 2721, 763, 220, 18, 92, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-single-call expected_match

Checks that did not pass for qwen3-xml/q38-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: single-call, reasoning, reasoning-prefilled.

Raw output

The user wants the weather in Paris. I should call get_weather.
</think>

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
<parameter=unit>
c
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-single-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "The user wants the weather in Paris. I should call get_weather.\n</think>\n\n<tool_call>\n<function=get_weather>\n<parameter=city>\nParis\n</parameter>\n<parameter=unit>\nc\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [760, 1156, 6587, 279, 8831, 303, 11751, 13, 353, 1220, 1562, 615, 67017, 13, 198, 248069, 271, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 57590, 198, 510, 15704, 29, 198, 27, 15704, 28, 3715, 29, 198, 66, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "The user wants the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-string-escapes expected_match

Checks that did not pass for qwen3-xml/q38-string-escapes
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Write it.', got 'Write it.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-string-escapes --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 15 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Write it.",
+  "content": "\n\n",
+  "reasoning_content": "Write it.\n",
   "tool_calls": [
     {

Fixture

Raw output

Write it.
</think>

<tool_call>
<function=write_file>
<parameter=path>
C:\tmp\x.txt
</parameter>
<parameter=content>
a "quote", a \backslash, <tag> & ampersand	tab
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write it.",
  "tool_calls": [
    {
      "arguments": {
        "content": "a \"quote\", a \\backslash, <tag> & ampersand\ttab",
        "path": "C:\\tmp\\x.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-string-escapes", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write it.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nC:\\tmp\\x.txt\n</parameter>\n<parameter=content>\na \"quote\", a \\backslash, <tag> & ampersand\ttab\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7734, 424, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 34, 6964, 5008, 3351, 3778, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 64, 328, 2855, 487, 264, 1088, 1371, 48569, 11, 361, 4429, 29, 594, 22050, 382, 429, 56196, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Write it.", "tool_calls": [{"name": "write_file", "arguments": {"path": "C:\\tmp\\x.txt", "content": "a \"quote\", a \\backslash, <tag> & ampersand\ttab"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "string-escapes"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. String values are inserted verbatim with no escaping."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-text-before-call expected_match

Checks that did not pass for qwen3-xml/q38-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got '\n\nLet me check the weather for you.\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the weather for you.",
-  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
+  "content": "\n\nLet me check the weather for you.\n\n",
+  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.\n",
   "tool_calls": [
     {

Fixture

Raw output

The user wants the weather in Paris. I should call get_weather.
</think>

Let me check the weather for you.

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-text-before-call", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "The user wants the weather in Paris. I should call get_weather.\n</think>\n\nLet me check the weather for you.\n\n<tool_call>\n<function=get_weather>\n<parameter=city>\nParis\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [760, 1156, 6587, 279, 8831, 303, 11751, 13, 353, 1220, 1562, 615, 67017, 13, 198, 248069, 271, 9764, 728, 1716, 279, 8831, 364, 488, 13, 271, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 57590, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": "Let me check the weather for you.", "reasoning_content": "The user wants the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "text-before-call"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Qwen3.5+ templates join content and the first <tool_call> with '\\n\\n', which is markup."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-thinking-disabled-parallel expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/q38-thinking-disabled-parallel
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-thinking-disabled-parallel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 19 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: parallel-calls, empty-arguments.

Raw output

<tool_call>
<function=get_weather>
<parameter=city>
Oslo
</parameter>
</function>
</tool_call>
<tool_call>
<function=get_time>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-thinking-disabled-parallel", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "<tool_call>\n<function=get_weather>\n<parameter=city>\nOslo\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=get_time>\n</function>\n</tool_call>", "output_token_ids": [248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 27925, 379, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 198, 248058, 198, 27, 1628, 27362, 2914, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Oslo"}}, {"name": "get_time", "arguments": {}}]}, "tags": ["parallel-calls", "empty-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      198,
      248068,
      271,
      248069,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-three-parallel-calls expected_match, stream_equals_nonstream

Checks that did not pass for qwen3-xml/q38-three-parallel-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n\n\n'; reasoning_content (whitespace only): expected 'Three cities, three calls.', got 'Three cities, three calls.\n'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream '\n\n', got '\n\n\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-three-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Three cities, three calls.",
+  "content": "\n\n",
+  "reasoning_content": "Three cities, three calls.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Three cities, three calls.",
+  "content": "\n\n\n\n",
+  "reasoning_content": "Three cities, three calls.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: parallel-calls, reasoning, reasoning-prefilled.

Raw output

Three cities, three calls.
</think>

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>
<tool_call>
<function=get_weather>
<parameter=city>
Berlin
</parameter>
</function>
</tool_call>
<tool_call>
<function=get_weather>
<parameter=city>
Tokyo
</parameter>
<parameter=unit>
f
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Three cities, three calls.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "f"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-three-parallel-calls", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Three cities, three calls.\n</think>\n\n<tool_call>\n<function=get_weather>\n<parameter=city>\nParis\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=get_weather>\n<parameter=city>\nBerlin\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=get_weather>\n<parameter=city>\nTokyo\n</parameter>\n<parameter=unit>\nf\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [19041, 9432, 11, 2250, 6526, 13, 198, 248069, 271, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 57590, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 198, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 91149, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059, 198, 248058, 198, 27, 1628, 27362, 67017, 29, 198, 27, 15704, 28, 8656, 29, 198, 51076, 15560, 198, 510, 15704, 29, 198, 27, 15704, 28, 3715, 29, 198, 69, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Three cities, three calls.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Berlin"}}, {"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "f"}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-truncated-at-tool-call-open expected_match

Checks that did not pass for qwen3-xml/q38-truncated-at-tool-call-open
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'The user wants the weather in Paris. I should call get_weather.', got 'The user wants the weather in Paris. I should call get_weather.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-truncated-at-tool-call-open --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 23 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
+  "content": "\n\n",
+  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: truncated, reasoning, reasoning-prefilled, x-stop-at-open-marker.

Raw output

The user wants the weather in Paris. I should call get_weather.
</think>

<tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the weather in Paris. I should call get_weather.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-truncated-at-tool-call-open", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "The user wants the weather in Paris. I should call get_weather.\n</think>\n\n<tool_call>", "output_token_ids": [760, 1156, 6587, 279, 8831, 303, 11751, 13, 353, 1220, 1562, 615, 67017, 13, 198, 248069, 271, 248058], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "The user wants the weather in Paris. I should call get_weather.", "tool_calls": []}, "tags": ["truncated", "reasoning", "reasoning-prefilled", "x-stop-at-open-marker"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Output stopped right after the <tool_call> opener. The reasoning is complete and must be kept; there is no call yet and the marker must not leak into content (SGLang #35565: https://github.com/sgl-project/sglang/issues/35565)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-unicode-emoji expected_match

Checks that did not pass for qwen3-xml/q38-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected '用户想知道东京的天气。🌦️', got '用户想知道东京的天气。🌦️\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-unicode-emoji --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "用户想知道东京的天气。🌦️",
+  "content": "\n\n",
+  "reasoning_content": "用户想知道东京的天气。🌦️\n",
   "tool_calls": [
     {

Fixture

Raw output

用户想知道东京的天气。🌦️
</think>

<tool_call>
<function=search>
<parameter=query>
東京の天気 ☀️🌧️ — «prévisions» für Zürich 🇨🇭
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "用户想知道东京的天气。🌦️",
  "tool_calls": [
    {
      "arguments": {
        "query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich 🇨🇭"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-unicode-emoji", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "用户想知道东京的天气。🌦️\n</think>\n\n<tool_call>\n<function=search>\n<parameter=query>\n東京の天気 ☀️🌧️ — «prévisions» für Zürich 🇨🇭\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [97237, 102487, 110799, 131802, 1710, 9008, 234, 99, 29545, 198, 248069, 271, 248058, 198, 27, 1628, 93260, 29, 198, 27, 15704, 76478, 29, 198, 115197, 15303, 247570, 24329, 222, 29545, 9008, 234, 100, 29545, 1892, 12129, 631, 933, 38685, 12620, 6954, 194323, 10838, 229, 101, 9008, 229, 255, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "用户想知道东京的天气。🌦️", "tool_calls": [{"name": "search", "arguments": {"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich 🇨🇭"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "unicode"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/q38-whitespace-significant expected_match

Checks that did not pass for qwen3-xml/q38-whitespace-significant
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Replace the block exactly.', got 'Replace the block exactly.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-whitespace-significant --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Replace the block exactly.",
+  "content": "\n\n",
+  "reasoning_content": "Replace the block exactly.\n",
   "tool_calls": [
     {

Fixture

Raw output

Replace the block exactly.
</think>

<tool_call>
<function=edit_file>
<parameter=path>
a.py
</parameter>
<parameter=old_string>
    if x:
        return 1

</parameter>
<parameter=new_string>
  two  spaces  
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Replace the block exactly.",
  "tool_calls": [
    {
      "arguments": {
        "new_string": "  two  spaces  ",
        "old_string": "    if x:\n        return 1\n",
        "path": "a.py"
      },
      "name": "edit_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/q38-whitespace-significant", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Replace the block exactly.\n</think>\n\n<tool_call>\n<function=edit_file>\n<parameter=path>\na.py\n</parameter>\n<parameter=old_string>\n    if x:\n        return 1\n\n</parameter>\n<parameter=new_string>\n  two  spaces  \n</parameter>\n</function>\n</tool_call>", "output_token_ids": [22386, 279, 2424, 6681, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 3468, 2378, 29, 198, 27, 15704, 79114, 29, 198, 64, 6971, 198, 510, 15704, 29, 198, 27, 15704, 28, 787, 3773, 29, 198, 262, 413, 830, 25, 198, 285, 460, 220, 16, 271, 510, 15704, 29, 198, 27, 15704, 8083, 3773, 29, 198, 220, 1330, 220, 12258, 2228, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Replace the block exactly.", "tool_calls": [{"name": "edit_file", "arguments": {"path": "a.py", "old_string": "    if x:\n        return 1\n", "new_string": "  two  spaces  "}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "x-whitespace-significant-string"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. Leading indentation and a trailing newline are part of the values. Exactly one '\\n' on each side of a value is markup (vLLM #48753: https://github.com/vllm-project/vllm/issues/48753)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "qwen3",
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3_5ForConditionalGeneration"
    ],
    "model_type": "qwen3_5"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3.8-27B",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      248045,
      74455,
      198,
      248068,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Qwen3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "qwen3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 248046,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3.8-27B",
    "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}

soft pass qwen3-xml/vllm-coder-missing-close-parameter expected_match

Checks that did not pass for qwen3-xml/vllm-coder-missing-close-parameter
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you:', got 'Let me check the weather for you:\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/vllm-coder-missing-close-parameter --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the weather for you:",
+  "content": "Let me check the weather for you:\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_qwen3coder_tool_parser.py#L932-L971. Tags: single-call, text-before-call, malformed, x-missing-close-param.

Raw output

Let me check the weather for you:
<tool_call>
<function=get_current_weather>
<parameter=city>
Dallas
<parameter=state>
TX
</parameter>
<parameter=unit>
fahrenheit
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "Let me check the weather for you:",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "qwen3-xml/vllm-coder-missing-close-parameter", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_qwen3coder_tool_parser.py#L932-L971", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city name"}, "state": {"type": "string", "description": "The state code"}, "unit": {"type": "string", "enum": ["fahrenheit", "celsius"]}}, "required": ["city", "state"]}, "description": "Get the current weather"}}, {"type": "function", "function": {"name": "calculate_area", "parameters": {"type": "object", "properties": {"shape": {"type": "string"}, "dimensions": {"type": "object"}, "precision": {"type": "integer"}}}, "description": "Calculate area of a shape"}}], "raw_output": "Let me check the weather for you:\n<tool_call>\n<function=get_current_weather>\n<parameter=city>\nDallas\n<parameter=state>\nTX\n</parameter>\n<parameter=unit>\nfahrenheit\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 9104, 369, 498, 510, 151657, 198, 27, 1688, 28280, 11080, 69364, 397, 27, 16181, 28, 8926, 397, 86514, 198, 27, 16181, 28, 2454, 397, 22867, 198, 522, 16181, 397, 27, 16181, 28, 3843, 397, 69, 47910, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "Let me check the weather for you:", "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}}]}, "tags": ["single-call", "text-before-call", "malformed", "x-missing-close-param"], "notes": "test_extract_tool_calls_missing_closing_parameter_tag: the next <parameter= ends the unclosed value."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "qwen3_coder"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Qwen3MoeForCausalLM"
    ],
    "model_type": "qwen3_moe"
  },
  "hf_config_error": null,
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151644,
      77091,
      198
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151645,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|im_end|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "Qwen2Tokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Qwen3CoderDetector",
  "tool_call_parser": "qwen3_coder",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model