CanIToolCall caniuse.com for tool calling

Qwen XML tool calls (Qwen3-Coder, Qwen3.5/3.6/3.8) on ollama 7af39318

fail 83% strict pass 45 pass · 9 fail

Run

Engine
ollama 7af39318 (7af393188defd52d370464de0d2064649cab9b41)
Run
to
Platform
linux-x86_64, Python 3.12.3, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "go_version": "go1.26.0",
  "llama_cpp_build": "b11081",
  "llama_cpp_commit": "161755f",
  "pinned_version": "7af393188defd52d370464de0d2064649cab9b41",
  "runner": "llama-server token rendering (preserved_tokens) + model/parsers via server/routes.go flow",
  "tokens_per_step": "one"
}
Results file
data/ollama-7af39318.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 44 0 9 0 83%
expected_error 1 0 0 0 100%
stream_equals_nonstream 52 0 2 0 96%
no_leakage 51 0 3 0 94%
arguments_json 42 0 0 0 100%
arguments_schema 42 0 0 0 100%
parallel_order 6 0 1 0 86%

10 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail qwen3-xml/bug-coder-missing-tool-call-opener expected_match, no_leakage
Checks that did not pass for qwen3-xml/bug-coder-missing-tool-call-opener
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para...; tool_calls: expected ['glob'], got []
expected_matchtoken failcontent: expected 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.', got 'I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<para...; tool_calls: expected ['glob'], got []
no_leakagenonstream failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'
no_leakagetoken failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/bug-coder-missing-tool-call-opener --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 18 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.",
+  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<parameter=pattern>\n**/*.md\n</parameter>\n</function>\n</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "pattern": "**/*.md"
-      },
-      "name": "glob"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18530. Tags: single-call, text-before-call, malformed, regression, x-bare-function.

Raw output

I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.

<function=glob>
<parameter=pattern>
**/*.md
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "pattern": "**/*.md"
      },
      "name": "glob"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/bug-coder-missing-tool-call-opener", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18530", "revision": "issue-18530", "license": "NOASSERTION", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py"}, "tools": [{"type": "function", "function": {"name": "glob", "parameters": {"type": "object", "properties": {"pattern": {"type": "string"}}, "required": ["pattern"]}, "description": "Find files by glob pattern"}}], "raw_output": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.\n\n<function=glob>\n<parameter=pattern>\n**/*.md\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [40, 686, 1156, 1779, 369, 678, 73192, 3542, 304, 279, 1482, 6220, 323, 1181, 1186, 75701, 11, 323, 1221, 1140, 1105, 1667, 279, 13206, 5392, 382, 27, 1688, 37162, 1684, 397, 27, 16181, 28, 14339, 397, 334, 23540, 2277, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "I will first check for all Markdown files in the current directory and its subdirectories, and then list them using the glob tool.", "reasoning_content": null, "tool_calls": [{"name": "glob", "arguments": {"pattern": "**/*.md"}}]}, "tags": ["single-call", "text-before-call", "malformed", "regression", "x-bare-function"], "notes": "A real qwen3-coder:30b generation captured by the reporter with raw: true (temperature 0, seed 1): the model omitted the <tool_call> opener after a prose preamble."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
fail qwen3-xml/bug-coder-number-outside-int64 expected_match
Checks that did not pass for qwen3-xml/bug-coder-number-outside-int64
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"x": 1e+20}', got '{"x":-9223372036854775808}'
expected_matchtoken failtool_calls[0].arguments: expected '{"x": 1e+20}', got '{"x":-9223372036854775808}'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/bug-coder-number-outside-int64 --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 20 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "x": 1e+20
+        "x": -9223372036854775808
       },
       "name": "calculate"

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18421. Tags: single-call, numeric-arguments, regression.

Raw output

<tool_call><function=calculate>
<parameter=x>
1e20
</parameter>
</function></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1e+20
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/bug-coder-number-outside-int64", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18421", "revision": "issue-18421", "license": "NOASSERTION", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py"}, "tools": [{"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"x": {"type": "number"}}}}}], "raw_output": "<tool_call><function=calculate>\n<parameter=x>\n1e20\n</parameter>\n</function></tool_call>", "output_token_ids": [151657, 27, 1688, 28, 35597, 397, 27, 16181, 25387, 397, 16, 68, 17, 15, 198, 522, 16181, 397, 522, 1688, 29, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": 1e+20}}]}, "tags": ["single-call", "numeric-arguments", "regression"], "notes": "The issue's parser input: a number outside the int64 range must be preserved (expected 100000000000000000000, not 9223372036854775807)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
fail qwen3-xml/bug-missing-close-parameter-before-function expected_match
Checks that did not pass for qwen3-xml/bug-missing-close-parameter-before-function
CheckStrategyResultDetail
expected_matchnonstream failparser raised 'OllamaError: XML syntax error on line 5: element <parameter> closed by </function>'
expected_matchtoken failparser raised 'OllamaError: XML syntax error on line 5: element <parameter> closed by </function>'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/bug-missing-close-parameter-before-function --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 17 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,16 +1,6 @@
 {
   "content": null,
+  "exception": "OllamaError: XML syntax error on line 5: element <parameter> closed by </function>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "body": {
-          "airConJobMode": "AIR_CLEAN",
-          "monitoringEnabled": true,
-          "windStrength": "HIGH"
-        }
-      },
-      "name": "ThinQ_Connect"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/57699. Tags: single-call, malformed, nested-json, regression, x-missing-close-param.

Raw output

<tool_call>
<function=ThinQ_Connect>
<parameter=body>
{"airConJobMode": "AIR_CLEAN", "windStrength": "HIGH", "monitoringEnabled": true}
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "body": {
          "airConJobMode": "AIR_CLEAN",
          "monitoringEnabled": true,
          "windStrength": "HIGH"
        }
      },
      "name": "ThinQ_Connect"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/bug-missing-close-parameter-before-function", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/57699", "revision": "issue-57699", "license": "NOASSERTION", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py"}, "tools": [{"type": "function", "function": {"name": "ThinQ_Connect", "parameters": {"type": "object", "properties": {"body": {"type": "object"}}, "required": ["body"]}}}], "raw_output": "<tool_call>\n<function=ThinQ_Connect>\n<parameter=body>\n{\"airConJobMode\": \"AIR_CLEAN\", \"windStrength\": \"HIGH\", \"monitoringEnabled\": true}\n</function>\n</tool_call>", "output_token_ids": [248058, 198, 27, 1628, 28, 89882, 48, 14659, 2245, 29, 198, 27, 15704, 28, 2513, 29, 198, 4754, 1264, 1073, 11895, 3513, 763, 328, 55247, 76916, 487, 328, 18574, 51079, 763, 328, 87116, 487, 328, 31162, 286, 5290, 763, 804, 92, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n\n</think>\n\n", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "ThinQ_Connect", "arguments": {"body": {"airConJobMode": "AIR_CLEAN", "windStrength": "HIGH", "monitoringEnabled": true}}}]}, "tags": ["single-call", "malformed", "nested-json", "regression", "x-missing-close-param"], "notes": "Raw text from the issue (Qwen3.8-27B, BFCL live_simple_40-17-0, temperature 0). The last parameter is closed by </function> without </parameter>; the call is complete, so the value must be kept (vLLM returns {} non-streaming). The issue's tool schema is not shown; 'body' is typed as an object here. The thinking mode of the request is not stated; the fixture assumes the no-think prompt."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3.5",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3.5, qwen3.6; create/metadata.go qwen3_5",
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3.8-27B.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "7b5dcc22c704d56bf35ebc9cc40ce3a2f766890c968f5c8b6d34a3808680749f"
}
fail qwen3-xml/coder-marker-in-arguments expected_match
Checks that did not pass for qwen3-xml/coder-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failparser raised 'OllamaError: XML syntax error on line 7: element <parameter> closed by </function>'
expected_matchtoken failparser raised 'OllamaError: XML syntax error on line 7: element <parameter> closed by </function>'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/coder-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 32 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "OllamaError: XML syntax error on line 7: element <parameter> closed by </function>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Close with </function> then </tool_call>; open with <function=x>.",
-        "path": "x.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<tool_call>
<function=write_file>
<parameter=path>
x.md
</parameter>
<parameter=content>
Close with </function> then </tool_call>; open with <function=x>.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "Close with </function> then </tool_call>; open with <function=x>.",
        "path": "x.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/coder-marker-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/b2cff646eb4bb1d68355c01b18ae02e7cf42d120/chat_template.jinja", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "5a38bfa05833266240066aedc497decc9b00cc0d3e3b8cceea98cf530196ab06"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "<tool_call>\n<function=write_file>\n<parameter=path>\nx.md\n</parameter>\n<parameter=content>\nClose with </function> then </tool_call>; open with <function=x>.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [151657, 198, 27, 1688, 28, 4934, 2458, 397, 27, 16181, 81940, 397, 87, 21324, 198, 522, 16181, 397, 27, 16181, 28, 1796, 397, 7925, 448, 690, 1688, 29, 1221, 220, 151658, 26, 1787, 448, 366, 1688, 25387, 29816, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "x.md", "content": "Close with </function> then </tool_call>; open with <function=x>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3-Coder-30B-A3B-Instruct chat template. XML-looking markers inside a value, not at line starts. '</tool_call>' is an added token, so it appears as its marker id in output_token_ids."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
fail qwen3-xml/llamacpp-coder-bare-function expected_match, no_leakage
Checks that did not pass for qwen3-xml/llamacpp-coder-bare-function
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>'; tool_calls: expected ['special_function'], got []
expected_matchtoken failcontent: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>'; tool_calls: expected ['special_function'], got []
no_leakagenonstream failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'
no_leakagetoken failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-coder-bare-function --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 9 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3559-L3569. Tags: single-call, malformed, numeric-arguments, x-bare-function.

Raw output

<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/llamacpp-coder-bare-function", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3559-L3569", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [27, 1688, 14149, 2964, 9174, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "malformed", "numeric-arguments", "x-bare-function"], "notes": "'Some models skip the opening <tool_call> and go straight to <function=>' (also Ollama #17353)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
fail qwen3-xml/llamacpp-coder-bare-function-parallel expected_match, no_leakage, parallel_order
Checks that did not pass for qwen3-xml/llamacpp-coder-bare-function-parallel
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>'; tool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function_with_opt']
expected_matchtoken failcontent: expected None, got '<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>'; tool_calls: expected ['special_function', 'special_function_with_opt'], got ['special_function_with_opt']
no_leakagenonstream failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'
no_leakagetoken failcontent contains '</tool_call>'; content contains '<function='; content contains '</function>'; content contains '<parameter='; content contains '</parameter>'
parallel_ordernonstream failexpected 2 calls ['special_function', 'special_function_with_opt'], got 1 ['special_function_with_opt']
parallel_ordertoken failexpected 2 calls ['special_function', 'special_function_with_opt'], got 1 ['special_function_with_opt']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-coder-bare-function-parallel --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 10 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": null,
+  "content": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>",
   "reasoning_content": null,
   "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    },
     {
       "arguments": {

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3586-L3612. Tags: parallel-calls, malformed, numeric-arguments, x-bare-function.

Raw output

<function=special_function>
<parameter=arg1>
1
</parameter>
</function>
</tool_call>
<tool_call>
<function=special_function_with_opt>
<parameter=arg1>
1
</parameter>
<parameter=arg2>
2
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    },
    {
      "arguments": {
        "arg1": 1,
        "arg2": 2
      },
      "name": "special_function_with_opt"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/llamacpp-coder-bare-function-parallel", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L3586-L3612", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}, {"type": "function", "function": {"name": "special_function_with_opt", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}, "arg2": {"type": "integer", "description": "The optional arg."}}, "required": ["arg1"]}, "description": "I'm special but have optional stuff"}}], "raw_output": "<function=special_function>\n<parameter=arg1>\n1\n</parameter>\n</function>\n</tool_call>\n<tool_call>\n<function=special_function_with_opt>\n<parameter=arg1>\n1\n</parameter>\n<parameter=arg2>\n2\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [27, 1688, 14149, 2964, 9174, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 522, 1688, 397, 151658, 198, 151657, 198, 27, 1688, 14149, 2964, 9174, 6615, 15032, 397, 27, 16181, 28, 858, 16, 397, 16, 198, 522, 16181, 397, 27, 16181, 28, 858, 17, 397, 17, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}, {"name": "special_function_with_opt", "arguments": {"arg1": 1, "arg2": 2}}]}, "tags": ["parallel-calls", "malformed", "numeric-arguments", "x-bare-function"]}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
fail qwen3-xml/q38-marker-in-arguments expected_match
Checks that did not pass for qwen3-xml/q38-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path":"notes.md","content":"Close tags inline: "}'
expected_matchtoken failtool_calls[0].arguments: expected '{"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}', got '{"path":"notes.md","content":"Close tags inline: "}'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 7 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here.",
+        "content": "Close tags inline: ",
         "path": "notes.md"
       },

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: single-call, reasoning, reasoning-prefilled, marker-in-arguments.

Raw output

Write the note.
</think>

<tool_call>
<function=write_file>
<parameter=path>
notes.md
</parameter>
<parameter=content>
Close tags inline: </parameter> and </function> and <parameter=path> are text here.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the note.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here.",
        "path": "notes.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/q38-marker-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Write the note.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nnotes.md\n</parameter>\n<parameter=content>\nClose tags inline: </parameter> and </function> and <parameter=path> are text here.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7734, 279, 5020, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 17742, 20668, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 7675, 9212, 7147, 25, 672, 15704, 29, 321, 672, 1628, 29, 321, 361, 15704, 79114, 29, 513, 1414, 1532, 13, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Write the note.", "tool_calls": [{"name": "write_file", "arguments": {"path": "notes.md", "content": "Close tags inline: </parameter> and </function> and <parameter=path> are text here."}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. The value contains XML-looking markers that are not at a line start. The structural delimiter is '\\n</parameter>\\n', so a parser that splits on a bare '</parameter>' cuts the value short."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3.5",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3.5, qwen3.6; create/metadata.go qwen3_5",
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3.8-27B.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "7b5dcc22c704d56bf35ebc9cc40ce3a2f766890c968f5c8b6d34a3808680749f"
}
fail qwen3-xml/q38-tool-call-token-in-arguments expected_match, stream_equals_nonstream
Checks that did not pass for qwen3-xml/q38-tool-call-token-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failparser raised 'OllamaError: XML syntax error on line 7: unexpected EOF'
expected_matchtoken failparser raised 'OllamaError: XML syntax error on line 7: unexpected EOF'
stream_equals_nonstreamtoken failreasoning_content: nonstream None, got 'Document the format.'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/rendered.jsonl --id qwen3-xml/q38-tool-call-token-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 8 of fixtures/qwen3-xml/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
-  "reasoning_content": "Document the format.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks.",
-        "path": "fmt.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "exception": "OllamaError: XML syntax error on line 7: unexpected EOF",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "OllamaError: XML syntax error on line 7: unexpected EOF",
   "reasoning_content": "Document the format.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks.",
-        "path": "fmt.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja. Tags: single-call, reasoning, reasoning-prefilled, marker-in-arguments.

Raw output

Document the format.
</think>

<tool_call>
<function=write_file>
<parameter=path>
fmt.md
</parameter>
<parameter=content>
Qwen wraps calls in <tool_call> ... </tool_call> blocks.
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Document the format.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks.",
        "path": "fmt.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/q38-tool-call-token-in-arguments", "family": "qwen3-xml", "models": ["Qwen/Qwen3.8-27B", "Qwen/Qwen3.6-35B-A3B"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/chat_template.jinja", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/render_qwen3_xml.py", "template_sha256": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}, "sites": {"type": "array", "items": {"type": "string"}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "edit_file", "description": "Replace an exact string in a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "old_string": {"type": "string"}, "new_string": {"type": "string"}}, "required": ["path", "old_string", "new_string"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}, {"type": "function", "function": {"name": "lookup", "description": "Look up a record by its identifiers.", "parameters": {"type": "object", "properties": {"zip": {"type": "string"}, "account_id": {"type": "string"}, "flag": {"type": "string"}, "payload": {"type": "string"}}, "required": ["zip"]}}}], "raw_output": "Document the format.\n</think>\n\n<tool_call>\n<function=write_file>\n<parameter=path>\nfmt.md\n</parameter>\n<parameter=content>\nQwen wraps calls in <tool_call> ... </tool_call> blocks.\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [7288, 279, 3443, 13, 198, 248069, 271, 248058, 198, 27, 1628, 28, 4775, 2378, 29, 198, 27, 15704, 79114, 29, 198, 12143, 20668, 198, 510, 15704, 29, 198, 27, 15704, 28, 1733, 29, 198, 48, 16451, 38394, 6526, 303, 220, 248058, 2423, 220, 248059, 9714, 13, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Document the format.", "tool_calls": [{"name": "write_file", "arguments": {"path": "fmt.md", "content": "Qwen wraps calls in <tool_call> ... </tool_call> blocks."}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "marker-in-arguments"], "notes": "History render of the official Qwen/Qwen3.8-27B chat template. '<tool_call>' and '</tool_call>' are added tokens, so the tokenizer maps them to their marker ids even inside the argument value (see output_token_ids). Only the XML structure tells them apart."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3.5",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3.5, qwen3.6; create/metadata.go qwen3_5",
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3.8-27B.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "7b5dcc22c704d56bf35ebc9cc40ce3a2f766890c968f5c8b6d34a3808680749f"
}
fail qwen3-xml/vllm-coder-missing-close-parameter expected_match, stream_equals_nonstream
Checks that did not pass for qwen3-xml/vllm-coder-missing-close-parameter
CheckStrategyResultDetail
expected_matchnonstream failparser raised 'OllamaError: XML syntax error on line 11: element <parameter> closed by </function>'
expected_matchtoken failparser raised 'OllamaError: XML syntax error on line 11: element <parameter> closed by </function>'
stream_equals_nonstreamtoken failcontent: nonstream None, got 'Let me check the weather for you:'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/vllm-coder-missing-close-parameter --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 16 of fixtures/qwen3-xml/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special

--- expected
+++ observed
@@ -1,14 +1,6 @@
 {
-  "content": "Let me check the weather for you:",
+  "content": null,
+  "exception": "OllamaError: XML syntax error on line 11: element <parameter> closed by </function>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Dallas",
-        "state": "TX",
-        "unit": "fahrenheit"
-      },
-      "name": "get_current_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,14 +1,6 @@
 {
   "content": "Let me check the weather for you:",
+  "exception": "OllamaError: XML syntax error on line 11: element <parameter> closed by </function>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Dallas",
-        "state": "TX",
-        "unit": "fahrenheit"
-      },
-      "name": "get_current_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_qwen3coder_tool_parser.py#L932-L971. Tags: single-call, text-before-call, malformed, x-missing-close-param.

Raw output

Let me check the weather for you:
<tool_call>
<function=get_current_weather>
<parameter=city>
Dallas
<parameter=state>
TX
</parameter>
<parameter=unit>
fahrenheit
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": "Let me check the weather for you:",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/vllm-coder-missing-close-parameter", "family": "qwen3-xml", "models": ["Qwen/Qwen3-Coder-30B-A3B-Instruct", "Qwen/Qwen3-Coder-480B-A35B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_qwen3coder_tool_parser.py#L932-L971", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city name"}, "state": {"type": "string", "description": "The state code"}, "unit": {"type": "string", "enum": ["fahrenheit", "celsius"]}}, "required": ["city", "state"]}, "description": "Get the current weather"}}, {"type": "function", "function": {"name": "calculate_area", "parameters": {"type": "object", "properties": {"shape": {"type": "string"}, "dimensions": {"type": "object"}, "precision": {"type": "integer"}}}, "description": "Calculate area of a shape"}}], "raw_output": "Let me check the weather for you:\n<tool_call>\n<function=get_current_weather>\n<parameter=city>\nDallas\n<parameter=state>\nTX\n</parameter>\n<parameter=unit>\nfahrenheit\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [10061, 752, 1779, 279, 9104, 369, 498, 510, 151657, 198, 27, 1688, 28280, 11080, 69364, 397, 27, 16181, 28, 8926, 397, 86514, 198, 27, 16181, 28, 2454, 397, 22867, 198, 522, 16181, 397, 27, 16181, 28, 3843, 397, 69, 47910, 198, 522, 16181, 397, 522, 1688, 397, 151658], "tokenizer": {"repo": "Qwen/Qwen3-Coder-30B-A3B-Instruct", "revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n", "expected": {"content": "Let me check the weather for you:", "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}}]}, "tags": ["single-call", "text-before-call", "malformed", "x-missing-close-param"], "notes": "test_extract_tool_calls_missing_closing_parameter_tag: the next <parameter= ends the unclosed value."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": false,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3-coder",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3-coder, qwen3-coder-next",
  "preserved_tokens": [
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": null,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3-Coder-30B-A3B-Instruct.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "b2cff646eb4bb1d68355c01b18ae02e7cf42d120",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "835d0d2ad5a7f95016cdf13dffb330e8495936aac89fa4edf5e103c424ce9849"
}
pass qwen3-xml/llamacpp-q35-call-inside-think

Synthetic strategies (not counted): expected_match@rand:2:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/qwen3-xml/imported.jsonl --id qwen3-xml/llamacpp-q35-call-inside-think --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 6 of fixtures/qwen3-xml/imported.jsonl.

The results file does not include the observed parses for this fixture.

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2323-L2340. Tags: single-call, reasoning, reasoning-prefilled, malformed, x-call-inside-think.

Raw output

Need to inspect the current directory.
<tool_call>
<function=run_in_terminal>
<parameter=command>
pwd
</parameter>
</function>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Need to inspect the current directory.",
  "tool_calls": [
    {
      "arguments": {
        "command": "pwd"
      },
      "name": "run_in_terminal"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "qwen3-xml/llamacpp-q35-call-inside-think", "family": "qwen3-xml", "models": ["Qwen/Qwen3.5-9B"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L2323-L2340", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/qwen3-xml/import_qwen3_xml.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "run_in_terminal", "parameters": {"type": "object", "properties": {"command": {"type": "string", "description": "Shell command to run"}}, "required": ["command"]}, "description": "Run a shell command."}}], "raw_output": "Need to inspect the current directory.\n<tool_call>\n<function=run_in_terminal>\n<parameter=command>\npwd\n</parameter>\n</function>\n</tool_call>", "output_token_ids": [22916, 310, 23331, 279, 1428, 6025, 13, 198, 248058, 198, 27, 1628, 28, 5917, 1201, 62138, 29, 198, 27, 15704, 28, 5454, 29, 198, 24751, 198, 510, 15704, 29, 198, 510, 1628, 29, 198, 248059], "tokenizer": {"repo": "Qwen/Qwen3.5-9B", "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a", "mode": "hf"}, "generation_prompt": "<|im_start|>assistant\n<think>\n", "expected": {"content": null, "reasoning_content": "Need to inspect the current directory.", "tool_calls": [{"name": "run_in_terminal", "arguments": {"command": "pwd"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "malformed", "x-call-inside-think"], "notes": "The model starts a tool call without closing the prefilled <think> block. llama.cpp's test ('a tool call ends the prefilled thinking block, with or without a closing </think>') treats <tool_call> as ending the reasoning. This is the most-reported failure for the family: https://github.com/ggml-org/llama.cpp/issues/20837, https://github.com/vllm-project/vllm/issues/39056."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "qwen3.5",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): qwen3.5, qwen3.6; create/metadata.go qwen3_5",
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "Qwen--Qwen3.5-9B.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "ccfbc41329c3caba178c629bf95b67eab472cee6f1a271ba69ce059056edd896"
}