GLM (4.5, 4.6, 4.7, 5.x) on vllm 0.30.0

fail 65% strict pass 33 pass · 11 soft pass · 7 fail

Run

Engine

vllm 0.30.0

Run

2026-09-26T05:19:00Z to 2026-09-26T05:20:52Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}

Results file

data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	33	11	4	69%
`expected_error`	0	0	3	0%
`stream_equals_nonstream`	35	11	5	69%
`split_invariance`	50	1	0	98%
`no_leakage`	49	0	2	96%
`arguments_json`	41	0	4	91%
`arguments_schema`	41	0	4	91%
`parallel_order`	8	0	0	100%

Fixtures needing attention

fail glm/glm45-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema

Checks that did not pass for glm/glm45-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md"}'
`expected_match`	`one`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`special`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`token`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:1:8`	fail	content: expected None, got '\n; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:2:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:3:8`	fail	content: expected None, got '\n\n; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:4:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:5:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '\n; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '\n\n; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n</tool_call>'; reasoning_content (whitespace only): nonstream '\nWrite the format note to the docs file.', got 'Write the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`split_invariance`	`*`	soft pass	vs one: rand:1:8 (soft_pass): content (whitespace only): one '; keys go in <arg_key>.</arg_value>\n</tool_call>', got '\n; keys go in <arg_key>.</arg_value>\n</tool_call>' \| rand:3:8 (soft_pass): content (whitespace only): one '; keys go in <arg_key>.</arg_value>\n</tool_call>', got '\n\n; keys go in <arg_key>.</arg_value>\n</tool_call>'
`no_leakage`	`one`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`special`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`token`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:1:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:2:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:3:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:4:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:5:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_schema`	`nonstream`	fail	[0] write_file: $: 'content' is a required property
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,9 +1,8 @@
 {
   "content": null,
-  "reasoning_content": "Write the format note to the docs file.",
+  "reasoning_content": "\nWrite the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
         "path": "docs/format.md"
       },

Strategies: one, special, token, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
+  "content": "; keys go in <arg_key>.</arg_value>\n</tool_call>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and "
       },
       "name": "write_file"

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
+  "content": "\n; keys go in <arg_key>.</arg_value>\n</tool_call>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and "
       },
       "name": "write_file"

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
+  "content": "\n\n; keys go in <arg_key>.</arg_value>\n</tool_call>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and "
       },
       "name": "write_file"

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, marker-in-arguments, reasoning, x-glm45.

Raw output

<think>Write the format note to the docs file.</think>
<tool_call>write_file
<arg_key>path</arg_key>
<arg_value>docs/format.md</arg_value>
<arg_key>content</arg_key>
<arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the format note to the docs file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
        "path": "docs/format.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-marker-in-arguments", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Write the format note to the docs file.</think>\n<tool_call>write_file\n<arg_key>path</arg_key>\n<arg_value>docs/format.md</arg_value>\n<arg_key>content</arg_key>\n<arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 7984, 279, 3561, 5185, 311, 279, 26257, 1034, 13, 151351, 198, 151352, 4934, 2458, 198, 151356, 2343, 151357, 198, 151358, 14111, 91564, 21274, 151359, 198, 151356, 1796, 151357, 198, 151358, 26701, 1817, 1618, 304, 220, 151352, 323, 220, 151353, 26, 6894, 728, 304, 220, 151356, 13, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Write the format note to the docs file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}}]}, "tags": ["single-call", "marker-in-arguments", "reasoning", "x-glm45"], "notes": "The string value contains <tool_call>, </tool_call> and <arg_key> (tokenized as the marker tokens). It does not contain </arg_value>, so the value is still delimited unambiguously."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm45-ollama-content-after-call expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-ollama-content-after-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'after tool', got None
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got 'after tool'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got 'after tool'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/imported.jsonl --id glm/glm45-ollama-content-after-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "after tool",
+  "content": null,
   "reasoning_content": "thinking",
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "after tool",
  "reasoning_content": "thinking",
  "tool_calls": [
    {
      "arguments": {},
      "name": "test"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119. Tags: single-call, empty-arguments, reasoning, text-after-call, x-glm45.

Raw output

<think>thinking</think><tool_call>test</tool_call>after tool

Expected parse

{
  "content": "after tool",
  "reasoning_content": "thinking",
  "tool_calls": [
    {
      "arguments": {},
      "name": "test"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-ollama-content-after-call", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119", "revision": "7af393188defd52d370464de0d2064649cab9b41", "license": "MIT", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright (c) Ollama (MIT)"}, "tools": [{"type": "function", "function": {"name": "test", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>thinking</think><tool_call>test</tool_call>after tool", "output_token_ids": [151350, 81583, 151351, 151352, 1944, 151353, 10689, 5392], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "expected": {"content": "after tool", "reasoning_content": "thinking", "tool_calls": [{"name": "test", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning", "text-after-call", "x-glm45"], "notes": "From Ollama's GLM-4.6 parser test ('tool call with content after'). GLM-4.5/4.6 open <think> themselves, so this is a complete completion after the <|assistant|> generation prompt. The tool schema is a stand-in; the test declares no tools."}

Parser configuration

{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm45-truncated-after-arg-key expected_error

Checks that did not pass for glm/glm45-truncated-after-arg-key
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm45-truncated-after-arg-key --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: truncated, x-glm45.

Expected graceful failure: max_tokens hit between two arguments of an open call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
<arg_key>unit</arg_key>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-truncated-after-arg-key", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n<arg_key>unit</arg_key>\n", "output_token_ids": [198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151356, 3843, 151357, 198], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit between two arguments of an open call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm45"], "notes": "Token prefix of glm/glm45-thinking-disabled."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "tool_call><|user|>\nUse the tools to answer./nothink<|assistant|>\n<think></think>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm47-marker-in-arguments expected_match, stream_equals_nonstream, no_leakage, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md"}'
`expected_match`	`one`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`special`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`token`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:1:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:2:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:3:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:4:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`expected_match`	`rand:5:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and '
`no_leakage`	`one`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`special`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`token`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:1:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:2:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:3:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:4:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:5:8`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and ': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_schema`	`nonstream`	fail	[0] write_file: $: 'content' is a required property
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm47-render.jsonl --id glm/glm47-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/glm/glm47-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,5 +5,4 @@
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
         "path": "docs/format.md"
       },

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
+  "content": "; keys go in <arg_key>.</arg_value></tool_call>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and "
       },
       "name": "write_file"

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: single-call, marker-in-arguments, reasoning, x-glm47.

Raw output

Write the format note to the docs file.</think><tool_call>write_file<arg_key>path</arg_key><arg_value>docs/format.md</arg_value><arg_key>content</arg_key><arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the format note to the docs file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
        "path": "docs/format.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-marker-in-arguments", "family": "glm", "models": ["zai-org/GLM-4.7", "zai-org/GLM-4.7-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "Write the format note to the docs file.</think><tool_call>write_file<arg_key>path</arg_key><arg_value>docs/format.md</arg_value><arg_key>content</arg_key><arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value></tool_call>", "output_token_ids": [7984, 279, 3561, 5185, 311, 279, 26257, 1034, 13, 151351, 151352, 4934, 2458, 151356, 2343, 151357, 151358, 14111, 91564, 21274, 151359, 151356, 1796, 151357, 151358, 26701, 1817, 1618, 304, 220, 151352, 323, 220, 151353, 26, 6894, 728, 304, 220, 151356, 13, 151359, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Write the format note to the docs file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}}]}, "tags": ["single-call", "marker-in-arguments", "reasoning", "x-glm47"], "notes": "The string value contains <tool_call>, </tool_call> and <arg_key> (tokenized as the marker tokens). It does not contain </arg_value>, so the value is still delimited unambiguously."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|><think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.7",
  "model_rule": "GLM-(4\\.7|5)",
  "model_type": "glm4_moe",
  "parser_source": "docs/formats/glm.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "}</arg_value>...</tool_call><|user|>Use the tools to answer.<|assistant|><think>",
  "reasoning_parser": "glm47",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.7",
    "requested_mode": "auto",
    "resolved_revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm47",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm47-missing-last-close-arg-value expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-missing-last-close-arg-value
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin"}'
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-missing-last-close-arg-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/glm/truncated-malformed.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Berlin",
-        "unit": "celsius"
+        "city": "Berlin"
       },
       "name": "get_weather"

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Berlin",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": \"celsius"
       },
       "name": "get_weather"

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/57826. Tags: single-call, malformed, regression, x-glm47, x-missing-close-arg, x-derived.

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-missing-last-close-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/57826", "revision": "issue opened 2026-09-20", "license": "NOASSERTION", "generator": "scripts/fixtures/glm/build.py", "attribution": "Issue text: \"a tool call that omits the last closing arg_value tag yields non-streaming arguments without the trailing parameter.\""}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin", "unit": "celsius"}}]}, "tags": ["single-call", "malformed", "regression", "x-glm47", "x-missing-close-arg", "x-derived"], "notes": "raw_output is NOT quoted from the issue (it gives no GLM string): it is derived from glm/glm47-thinking-disabled by deleting the last </arg_value> token, the malformation #57826 describes. </tool_call> still closes the call, so the trailing value is recoverable; the issue (and the qwen3 fix it mirrors, vLLM #57707) expects the parser to keep it."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.7",
  "model_rule": "GLM-(4\\.7|5)",
  "model_type": "glm4_moe",
  "parser_source": "docs/formats/glm.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "</arg_value>...</tool_call><|user|>Use the tools to answer.<|assistant|></think>",
  "reasoning_parser": "glm47",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.7",
    "requested_mode": "auto",
    "resolved_revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm47",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm47-truncated-before-close-call expected_error

Checks that did not pass for glm/glm47-truncated-before-close-call
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-before-close-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit after the last </arg_value>; </tool_call> is missing. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-before-close-call", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151359], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit after the last </arg_value>; </tool_call> is missing.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled. A parser should not emit a call that never closed."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.7",
  "model_rule": "GLM-(4\\.7|5)",
  "model_type": "glm4_moe",
  "parser_source": "docs/formats/glm.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "</arg_value>...</tool_call><|user|>Use the tools to answer.<|assistant|></think>",
  "reasoning_parser": "glm47",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.7",
    "requested_mode": "auto",
    "resolved_revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm47",
  "units_source": "fixture.output_token_ids"
}

fail glm/glm47-truncated-inside-arg-value expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-truncated-inside-arg-value
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-inside-arg-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": \""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit inside the second <arg_value>; the call never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-inside-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit inside the second <arg_value>; the call never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.7",
  "model_rule": "GLM-(4\\.7|5)",
  "model_type": "glm4_moe",
  "parser_source": "docs/formats/glm.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "</arg_value>...</tool_call><|user|>Use the tools to answer.<|assistant|></think>",
  "reasoning_parser": "glm47",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.7",
    "requested_mode": "auto",
    "resolved_revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm47",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-call-order-vs-tool-order expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-call-order-vs-tool-order
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nWeather, then search, then weather again.', got 'Weather, then search, then weather again.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-call-order-vs-tool-order --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 12 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "reasoning_content": "\nWeather, then search, then weather again.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Weather, then search, then weather again.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "query": "Oslo museums"
      },
      "name": "search_web"
    },
    {
      "arguments": {
        "city": "Bergen"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, reasoning, x-call-index-vs-tool-index, x-glm45.

Raw output

<think>Weather, then search, then weather again.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Oslo</arg_value>
</tool_call>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>Oslo museums</arg_value>
</tool_call>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Bergen</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Weather, then search, then weather again.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "query": "Oslo museums"
      },
      "name": "search_web"
    },
    {
      "arguments": {
        "city": "Bergen"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-call-order-vs-tool-order", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Weather, then search, then weather again.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Oslo</arg_value>\n</tool_call>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>Oslo museums</arg_value>\n</tool_call>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Bergen</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 28876, 11, 1221, 2711, 11, 1221, 9101, 1549, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 28764, 385, 151359, 198, 151353, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 28764, 385, 50245, 151359, 198, 151353, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 33, 2375, 268, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Weather, then search, then weather again.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Oslo"}}, {"name": "search_web", "arguments": {"query": "Oslo museums"}}, {"name": "get_weather", "arguments": {"city": "Bergen"}}]}, "tags": ["parallel-calls", "reasoning", "x-call-index-vs-tool-index", "x-glm45"], "notes": "The same tool is called twice around another one; call indices must be 0,1,2, not the tools-list position (https://github.com/sgl-project/sglang/issues/33324)."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-content-before-call expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`one`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`special`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\n\nI'll look that up for you.\n"
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream "I'll look that up for you.", got "\n\nI'll look that up for you.\n"; reasoning_content (whitespace only): nonstream '\nNeed live data, so call the tool.', got 'Need live data, so call the tool.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-content-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look that up for you.",
+  "content": "\n\nI'll look that up for you.\n",
   "reasoning_content": "Need live data, so call the tool.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, text-before-call, reasoning, x-glm45.

Raw output

<think>Need live data, so call the tool.</think>
I'll look that up for you.
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll look that up for you.",
  "reasoning_content": "Need live data, so call the tool.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-content-before-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Need live data, so call the tool.</think>\nI'll look that up for you.\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 23591, 3887, 821, 11, 773, 1618, 279, 5392, 13, 151351, 198, 40, 3278, 1401, 429, 705, 369, 498, 624, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": "I'll look that up for you.", "reasoning_content": "Need live data, so call the tool.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "x-glm45"], "notes": "The template separates content from the call (GLM-4.5 with a newline); the correct content is the message text without that separator."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-empty-arguments expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nJust fetch the time.', got 'Just fetch the time.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Just fetch the time.",
+  "reasoning_content": "\nJust fetch the time.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Just fetch the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, empty-arguments, reasoning, x-glm45.

Raw output

<think>Just fetch the time.</think>
<tool_call>get_time
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Just fetch the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-empty-arguments", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Just fetch the time.</think>\n<tool_call>get_time\n</tool_call>", "output_token_ids": [198, 151350, 9947, 7807, 279, 882, 13, 151351, 198, 151352, 455, 3009, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Just fetch the time.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-nested-object expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-nested-object
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nApply the dark theme and disable SMS.', got 'Apply the dark theme and disable SMS.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-nested-object --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Apply the dark theme and disable SMS.",
+  "reasoning_content": "\nApply the dark theme and disable SMS.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Apply the dark theme and disable SMS.",
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}

Fixture

Raw output

<think>Apply the dark theme and disable SMS.</think>
<tool_call>update_settings
<arg_key>settings</arg_key>
<arg_value>{"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}</arg_value>
<arg_key>dry_run</arg_key>
<arg_value>false</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Apply the dark theme and disable SMS.",
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-nested-object", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Apply the dark theme and disable SMS.</think>\n<tool_call>update_settings\n<arg_key>settings</arg_key>\n<arg_value>{\"theme\": {\"mode\": \"dark\", \"accent\": [255, 128, 0]}, \"notifications\": {\"email\": true, \"sms\": false}}</arg_value>\n<arg_key>dry_run</arg_key>\n<arg_value>false</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 28397, 279, 6319, 6912, 323, 11151, 29793, 13, 151351, 198, 151352, 2386, 10848, 198, 151356, 6511, 151357, 198, 151358, 4913, 9044, 788, 5212, 8515, 788, 330, 22565, 497, 330, 76928, 788, 508, 122300, 11, 220, 109295, 11, 220, 15, 65643, 330, 37985, 788, 5212, 2332, 788, 830, 11, 330, 41158, 788, 895, 3417, 151359, 198, 151356, 52857, 13998, 151357, 198, 151358, 3849, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Apply the dark theme and disable SMS.", "tool_calls": [{"name": "update_settings", "arguments": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}]}, "tags": ["single-call", "nested-json", "numeric-arguments", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-no-call expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\n\nParis is the capital of France.'
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream '\nParis is the capital of France.', got '\n\nParis is the capital of France.'; reasoning_content (whitespace only): nonstream '\nA simple fact; no tool is needed.', got 'A simple fact; no tool is needed.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Paris is the capital of France.",
-  "reasoning_content": "A simple fact; no tool is needed.",
+  "content": "\nParis is the capital of France.",
+  "reasoning_content": "\nA simple fact; no tool is needed.",
   "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Paris is the capital of France.",
+  "content": "\n\nParis is the capital of France.",
   "reasoning_content": "A simple fact; no tool is needed.",
   "tool_calls": []

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: no-call, reasoning, x-glm45.

Raw output

<think>A simple fact; no tool is needed.</think>
Paris is the capital of France.

Expected parse

{
  "content": "Paris is the capital of France.",
  "reasoning_content": "A simple fact; no tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-no-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>A simple fact; no tool is needed.</think>\nParis is the capital of France.", "output_token_ids": [198, 151350, 32, 4285, 2097, 26, 902, 5392, 374, 4362, 13, 151351, 198, 59190, 374, 279, 6722, 315, 9621, 13], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": "Paris is the capital of France.", "reasoning_content": "A simple fact; no tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-parallel-mixed-nested expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-parallel-mixed-nested
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nCheck the weather, then search for cafés.', got 'Check the weather, then search for cafés.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-parallel-mixed-nested --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "reasoning_content": "\nCheck the weather, then search for cafés.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Check the weather, then search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max_results": 5,
          "site": "example.com",
          "tags": [
            "coffee",
            "wifi"
          ]
        },
        "query": "best cafés"
      },
      "name": "search_web"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, nested-json, reasoning, x-glm45.

Raw output

<think>Check the weather, then search for cafés.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Zürich</arg_value>
<arg_key>unit</arg_key>
<arg_value>celsius</arg_value>
</tool_call>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>best cafés</arg_value>
<arg_key>filters</arg_key>
<arg_value>{"site": "example.com", "max_results": 5, "tags": ["coffee", "wifi"]}</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Check the weather, then search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max_results": 5,
          "site": "example.com",
          "tags": [
            "coffee",
            "wifi"
          ]
        },
        "query": "best cafés"
      },
      "name": "search_web"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-parallel-mixed-nested", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Check the weather, then search for cafés.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Zürich</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>celsius</arg_value>\n</tool_call>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>best cafés</arg_value>\n<arg_key>filters</arg_key>\n<arg_value>{\"site\": \"example.com\", \"max_results\": 5, \"tags\": [\"coffee\", \"wifi\"]}</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 3973, 279, 9101, 11, 1221, 2711, 369, 28996, 5397, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 57, 5186, 713, 151359, 198, 151356, 3843, 151357, 198, 151358, 66, 40026, 151359, 198, 151353, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 15850, 28996, 5397, 151359, 198, 151356, 24533, 151357, 198, 151358, 4913, 9509, 788, 330, 8686, 905, 497, 330, 2810, 13567, 788, 220, 20, 11, 330, 14073, 788, 4383, 77489, 497, 330, 66791, 91669, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Check the weather, then search for cafés.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "celsius"}}, {"name": "search_web", "arguments": {"query": "best cafés", "filters": {"site": "example.com", "max_results": 5, "tags": ["coffee", "wifi"]}}}]}, "tags": ["parallel-calls", "nested-json", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-parallel-same-tool expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-parallel-same-tool
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nTwo cities, so two independent weather lookups.', got 'Two cities, so two independent weather lookups.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-parallel-same-tool --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "reasoning_content": "\nTwo cities, so two independent weather lookups.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, reasoning, x-glm45.

Raw output

<think>Two cities, so two independent weather lookups.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Paris</arg_value>
</tool_call>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Tokyo</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-parallel-same-tool", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Two cities, so two independent weather lookups.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Paris</arg_value>\n</tool_call>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Tokyo</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 11608, 9716, 11, 773, 1378, 9485, 9101, 1401, 8601, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 59190, 151359, 198, 151353, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 52504, 16019, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Two cities, so two independent weather lookups.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["parallel-calls", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-schema-coercion expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-schema-coercion
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThree second timer labelled 3.', got 'Three second timer labelled 3.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-schema-coercion --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Three second timer labelled 3.",
+  "reasoning_content": "\nThree second timer labelled 3.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Three second timer labelled 3.",
  "tool_calls": [
    {
      "arguments": {
        "label": "3",
        "seconds": 3
      },
      "name": "set_timer"
    }
  ]
}

Fixture

Raw output

<think>Three second timer labelled 3.</think>
<tool_call>set_timer
<arg_key>seconds</arg_key>
<arg_value>3</arg_value>
<arg_key>label</arg_key>
<arg_value>3</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Three second timer labelled 3.",
  "tool_calls": [
    {
      "arguments": {
        "label": "3",
        "seconds": 3
      },
      "name": "set_timer"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-schema-coercion", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Three second timer labelled 3.</think>\n<tool_call>set_timer\n<arg_key>seconds</arg_key>\n<arg_value>3</arg_value>\n<arg_key>label</arg_key>\n<arg_value>3</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 19604, 2086, 9018, 59817, 220, 18, 13, 151351, 198, 151352, 746, 16241, 198, 151356, 17381, 151357, 198, 151358, 18, 151359, 198, 151356, 1502, 151357, 198, 151358, 18, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Three second timer labelled 3.", "tool_calls": [{"name": "set_timer", "arguments": {"seconds": 3, "label": "3"}}]}, "tags": ["single-call", "numeric-arguments", "x-schema-coercion", "reasoning", "x-glm45"], "notes": "Both values render as the bare text 3; only the tool schema tells the integer from the string."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-single-call expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream '\nThe user wants the current weather in Paris, in Celsius.', got 'The user wants the current weather in Paris, in Celsius.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
+  "reasoning_content": "\nThe user wants the current weather in Paris, in Celsius.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, reasoning, x-glm45.

Raw output

<think>The user wants the current weather in Paris, in Celsius.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Paris</arg_value>
<arg_key>unit</arg_key>
<arg_value>celsius</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-single-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>The user wants the current weather in Paris, in Celsius.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Paris</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>celsius</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 785, 1196, 6801, 279, 1482, 9101, 304, 12089, 11, 304, 60916, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 59190, 151359, 198, 151356, 3843, 151357, 198, 151358, 66, 40026, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris, in Celsius.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "celsius"}}]}, "tags": ["single-call", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-unicode-emoji expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`one`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`special`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`rand:1:8`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`rand:2:8`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`rand:3:8`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`rand:4:8`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."
`stream_equals_nonstream`	`rand:5:8`	soft pass	reasoning_content (whitespace only): nonstream "\nSearch with the user's exact wording.", got "Search with the user's exact wording."

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-unicode-emoji --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Search with the user's exact wording.",
+  "reasoning_content": "\nSearch with the user's exact wording.",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Search with the user's exact wording.",
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, unicode, reasoning, x-glm45.

Raw output

<think>Search with the user's exact wording.</think>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>Zürich café ☕ 東京の天気 🌸</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Search with the user's exact wording.",
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-unicode-emoji", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Search with the user's exact wording.</think>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>Zürich café ☕ 東京の天気 🌸</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 5890, 448, 279, 1196, 594, 4734, 59808, 13, 151351, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 57, 5186, 713, 51609, 25048, 243, 60173, 109, 98971, 15755, 98411, 128426, 11157, 234, 116, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Search with the user's exact wording.", "tool_calls": [{"name": "search_web", "arguments": {"query": "Zürich café ☕ 東京の天気 🌸"}}]}, "tags": ["single-call", "unicode", "reasoning", "x-glm45"]}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "ue-2}</arg_value>\n...\n</tool_call><|user|>\nUse the tools to answer.<|assistant|>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

soft pass glm/glm45-vllm-newline-format-content-before expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-vllm-newline-format-content-before
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`special`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-newline-format-content-before --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": "I'll check it.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll check it.",
+  "content": "I'll check it. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137. Tags: single-call, text-before-call, x-whitespace-variant-45, x-glm45.

Raw output

I'll check it. <tool_call>get_current_weather
<arg_key>city</arg_key>
<arg_value>Dallas</arg_value>
<arg_key>state</arg_key>
<arg_value>TX</arg_value>
<arg_key>unit</arg_key>
<arg_value>fahrenheit</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll check it.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-newline-format-content-before", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "I'll check it. <tool_call>get_current_weather\n<arg_key>city</arg_key>\n<arg_value>Dallas</arg_value>\n<arg_key>state</arg_key>\n<arg_value>TX</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>fahrenheit</arg_value>\n</tool_call>", "output_token_ids": [40, 3278, 1779, 432, 13, 220, 151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 198, 151358, 85796, 151359, 198, 151356, 2454, 151357, 198, 151358, 22806, 151359, 198, 151356, 3843, 151357, 198, 151358, 69, 47609, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": "I'll check it.", "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}}]}, "tags": ["single-call", "text-before-call", "x-whitespace-variant-45", "x-glm45"], "notes": "vLLM expects the separator space before <tool_call> to be dropped from content."}

Parser configuration

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "zai-org/GLM-4.5",
  "model_rule": "GLM-4\\.[56]",
  "model_type": "glm4_moe",
  "parser_source": "GLM-4.5/4.6 model cards: --tool-call-parser glm45 --reasoning-parser glm45",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "tool_call><|user|>\nUse the tools to answer./nothink<|assistant|>\n<think></think>",
  "reasoning_parser": "glm45",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "zai-org/GLM-4.5",
    "requested_mode": "auto",
    "resolved_revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "glm45",
  "units_source": "fixture.output_token_ids"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model