GLM (4.5, 4.6, 4.7, 5.x) on sglang 0.5.20

fail 49% strict pass 25 pass · 2 soft pass · 24 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	25	2	21	52%
`expected_error`	0	0	3	0%
`stream_equals_nonstream`	26	2	23	51%
`split_invariance`	32	7	12	63%
`no_leakage`	39	0	12	76%
`arguments_json`	38	0	6	86%
`arguments_schema`	38	0	6	86%
`parallel_order`	0	0	8	0%

Fixtures needing attention

fail glm/glm45-call-order-vs-tool-order expected_match, stream_equals_nonstream, split_invariance, no_leakage, parallel_order

Checks that did not pass for glm/glm45-call-order-vs-tool-order
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Weather, then search, then weather again.', got '\n<think>Weather, then search, then weather again.'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'; tool_calls: expected ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'; tool_calls: expected ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n\n'; reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'; tool_calls: nonstream ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'; tool_calls: nonstream ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n\n\n'; reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`rand:3:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>Weather, then search, then weather again.', got '\nWeather, then search, then weather again.'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:3:8 (fail): content (whitespace only): one None, got '\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather']
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'search_web', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 3 calls ['get_weather', 'search_web', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-call-order-vs-tool-order --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "reasoning_content": "\n<think>Weather, then search, then weather again.",
   "tool_calls": [
     {

Strategies: one, special

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "reasoning_content": "\nWeather, then search, then weather again.",
   "tool_calls": [
     {
@@ -8,16 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "query": "Oslo museums"
-      },
-      "name": "search_web"
-    },
-    {
-      "arguments": {
-        "city": "Bergen"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "content": "\n\n\n",
+  "reasoning_content": "\nWeather, then search, then weather again.",
   "tool_calls": [
     {

Strategies: rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "reasoning_content": "\nWeather, then search, then weather again.",
   "tool_calls": [
     {

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Weather, then search, then weather again.",
+  "content": "\n",
+  "reasoning_content": "\nWeather, then search, then weather again.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, reasoning, x-call-index-vs-tool-index, x-glm45.

Raw output

<think>Weather, then search, then weather again.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Oslo</arg_value>
</tool_call>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>Oslo museums</arg_value>
</tool_call>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Bergen</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Weather, then search, then weather again.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "query": "Oslo museums"
      },
      "name": "search_web"
    },
    {
      "arguments": {
        "city": "Bergen"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-call-order-vs-tool-order", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Weather, then search, then weather again.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Oslo</arg_value>\n</tool_call>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>Oslo museums</arg_value>\n</tool_call>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Bergen</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 28876, 11, 1221, 2711, 11, 1221, 9101, 1549, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 28764, 385, 151359, 198, 151353, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 28764, 385, 50245, 151359, 198, 151353, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 33, 2375, 268, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Weather, then search, then weather again.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Oslo"}}, {"name": "search_web", "arguments": {"query": "Oslo museums"}}, {"name": "get_weather", "arguments": {"city": "Bergen"}}]}, "tags": ["parallel-calls", "reasoning", "x-call-index-vs-tool-index", "x-glm45"], "notes": "The same tool is called twice around another one; call indices must be 0,1,2, not the tools-list position (https://github.com/sgl-project/sglang/issues/33324)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-content-before-call expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Need live data, so call the tool.', got '\n<think>Need live data, so call the tool.'
`expected_match`	`one`	fail	content: expected "I'll look that up for you.", got None; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`special`	fail	content: expected "I'll look that up for you.", got None; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\nI'll look that up for you.\n"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`rand:1:8`	fail	content: expected "I'll look that up for you.", got "\nI'll look that up for you"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`rand:2:8`	fail	content: expected "I'll look that up for you.", got "\nI'll look that up for"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "\nI'll look that up for you.\n"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`rand:4:8`	fail	content: expected "I'll look that up for you.", got "\nI'll look that up"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`expected_match`	`rand:5:8`	fail	content: expected "I'll look that up for you.", got "\nI'll look that up for you"; reasoning_content (whitespace only): expected 'Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`one`	fail	content: nonstream "I'll look that up for you.", got None; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`special`	fail	content: nonstream "I'll look that up for you.", got None; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream "I'll look that up for you.", got "\nI'll look that up for you.\n"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream "I'll look that up for you.", got "\nI'll look that up for you"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream "I'll look that up for you.", got "\nI'll look that up for"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`rand:3:8`	fail	content (whitespace only): nonstream "I'll look that up for you.", got "\nI'll look that up for you.\n"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream "I'll look that up for you.", got "\nI'll look that up"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream "I'll look that up for you.", got "\nI'll look that up for you"; reasoning_content: nonstream '\n<think>Need live data, so call the tool.', got '\nNeed live data, so call the tool.'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got "\nI'll look that up for you.\n" \| rand:1:8 (fail): content: one None, got "\nI'll look that up for you" \| rand:2:8 (fail): content: one None, got "\nI'll look that up for" \| rand:3:8 (fail): content: one None, got "\nI'll look that up for you.\n" \| rand:4:8 (fail): content: one None, got "\nI'll look that up" \| rand:5:8 (fail): content: one None, got "\nI'll look that up for you"
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-content-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "reasoning_content": "\n<think>Need live data, so call the tool.",
   "tool_calls": [
     {

Strategies: one, special

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "content": null,
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "content": "\nI'll look that up for you.\n",
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "content": "\nI'll look that up for you",
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "content": "\nI'll look that up for",
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "I'll look that up for you.",
-  "reasoning_content": "Need live data, so call the tool.",
+  "content": "\nI'll look that up",
+  "reasoning_content": "\nNeed live data, so call the tool.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, text-before-call, reasoning, x-glm45.

Raw output

<think>Need live data, so call the tool.</think>
I'll look that up for you.
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll look that up for you.",
  "reasoning_content": "Need live data, so call the tool.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-content-before-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Need live data, so call the tool.</think>\nI'll look that up for you.\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 23591, 3887, 821, 11, 773, 1618, 279, 5392, 13, 151351, 198, 40, 3278, 1401, 429, 705, 369, 498, 624, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": "I'll look that up for you.", "reasoning_content": "Need live data, so call the tool.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "x-glm45"], "notes": "The template separates content from the call (GLM-4.5 with a newline); the correct content is the message text without that separator."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-empty-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Just fetch the time.', got '\n<think>Just fetch the time.'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`rand:4:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>Just fetch the time.', got '\nJust fetch the time.'
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:4:8 (soft_pass): content (whitespace only): one None, got '\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Just fetch the time.",
+  "reasoning_content": "\n<think>Just fetch the time.",
   "tool_calls": [
     {

Strategies: one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Just fetch the time.",
+  "reasoning_content": "\nJust fetch the time.",
   "tool_calls": [
     {

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Just fetch the time.",
+  "content": "\n",
+  "reasoning_content": "\nJust fetch the time.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, empty-arguments, reasoning, x-glm45.

Raw output

<think>Just fetch the time.</think>
<tool_call>get_time
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Just fetch the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-empty-arguments", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Just fetch the time.</think>\n<tool_call>get_time\n</tool_call>", "output_token_ids": [198, 151350, 9947, 7807, 279, 882, 13, 151351, 198, 151352, 455, 3009, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Just fetch the time.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema

Checks that did not pass for glm/glm45-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Write the format note to the docs file.', got '\n<think>Write the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md"}'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`token`	fail	content: expected None, got '\n; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:1:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:2:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:3:8`	fail	content: expected None, got '\n; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:4:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:5:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content (whitespace only): expected 'Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got '\n; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got '\n; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got '; keys go in <arg_key>.</arg_value>\n'; reasoning_content: nonstream '\n<think>Write the format note to the docs file.', got '\nWrite the format note to the docs file.'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got '\n; keys go in <arg_key>.</arg_value>\n' \| rand:1:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>\n' \| rand:2:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>\n' \| rand:3:8 (fail): content: one None, got '\n; keys go in <arg_key>.</arg_value>\n' \| rand:4:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>\n' \| rand:5:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'
`no_leakage`	`token`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:1:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:2:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:3:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:4:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:5:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_schema`	`nonstream`	fail	[0] write_file: $: 'content' is a required property
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,9 +1,8 @@
 {
   "content": null,
-  "reasoning_content": "Write the format note to the docs file.",
+  "reasoning_content": "\n<think>Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
         "path": "docs/format.md"
       },

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
   "content": null,
-  "reasoning_content": "Write the format note to the docs file.",
+  "reasoning_content": "\nWrite the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and}"
       },
       "name": "write_file"

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
-  "reasoning_content": "Write the format note to the docs file.",
+  "content": "\n; keys go in <arg_key>.</arg_value>\n",
+  "reasoning_content": "\nWrite the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and}"
       },
       "name": "write_file"

Strategies: rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
-  "reasoning_content": "Write the format note to the docs file.",
+  "content": "; keys go in <arg_key>.</arg_value>\n",
+  "reasoning_content": "\nWrite the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and}"
       },
       "name": "write_file"

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, marker-in-arguments, reasoning, x-glm45.

Raw output

<think>Write the format note to the docs file.</think>
<tool_call>write_file
<arg_key>path</arg_key>
<arg_value>docs/format.md</arg_value>
<arg_key>content</arg_key>
<arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the format note to the docs file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
        "path": "docs/format.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-marker-in-arguments", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Write the format note to the docs file.</think>\n<tool_call>write_file\n<arg_key>path</arg_key>\n<arg_value>docs/format.md</arg_value>\n<arg_key>content</arg_key>\n<arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 7984, 279, 3561, 5185, 311, 279, 26257, 1034, 13, 151351, 198, 151352, 4934, 2458, 198, 151356, 2343, 151357, 198, 151358, 14111, 91564, 21274, 151359, 198, 151356, 1796, 151357, 198, 151358, 26701, 1817, 1618, 304, 220, 151352, 323, 220, 151353, 26, 6894, 728, 304, 220, 151356, 13, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Write the format note to the docs file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}}]}, "tags": ["single-call", "marker-in-arguments", "reasoning", "x-glm45"], "notes": "The string value contains <tool_call>, </tool_call> and <arg_key> (tokenized as the marker tokens). It does not contain </arg_value>, so the value is still delimited unambiguously."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-nested-object expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-nested-object
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Apply the dark theme and disable SMS.', got '\n<think>Apply the dark theme and disable SMS.'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:1:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`stream_equals_nonstream`	`rand:5:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Apply the dark theme and disable SMS.', got '\nApply the dark theme and disable SMS.'
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:1:8 (soft_pass): content (whitespace only): one None, got '\n' \| rand:5:8 (soft_pass): content (whitespace only): one None, got '\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-nested-object --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Apply the dark theme and disable SMS.",
+  "reasoning_content": "\n<think>Apply the dark theme and disable SMS.",
   "tool_calls": [
     {

Strategies: one, special, rand:2:8, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Apply the dark theme and disable SMS.",
+  "reasoning_content": "\nApply the dark theme and disable SMS.",
   "tool_calls": [
     {

Strategies: token, rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Apply the dark theme and disable SMS.",
+  "content": "\n",
+  "reasoning_content": "\nApply the dark theme and disable SMS.",
   "tool_calls": [
     {

Fixture

Raw output

<think>Apply the dark theme and disable SMS.</think>
<tool_call>update_settings
<arg_key>settings</arg_key>
<arg_value>{"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}</arg_value>
<arg_key>dry_run</arg_key>
<arg_value>false</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Apply the dark theme and disable SMS.",
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-nested-object", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Apply the dark theme and disable SMS.</think>\n<tool_call>update_settings\n<arg_key>settings</arg_key>\n<arg_value>{\"theme\": {\"mode\": \"dark\", \"accent\": [255, 128, 0]}, \"notifications\": {\"email\": true, \"sms\": false}}</arg_value>\n<arg_key>dry_run</arg_key>\n<arg_value>false</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 28397, 279, 6319, 6912, 323, 11151, 29793, 13, 151351, 198, 151352, 2386, 10848, 198, 151356, 6511, 151357, 198, 151358, 4913, 9044, 788, 5212, 8515, 788, 330, 22565, 497, 330, 76928, 788, 508, 122300, 11, 220, 109295, 11, 220, 15, 65643, 330, 37985, 788, 5212, 2332, 788, 830, 11, 330, 41158, 788, 895, 3417, 151359, 198, 151356, 52857, 13998, 151357, 198, 151358, 3849, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Apply the dark theme and disable SMS.", "tool_calls": [{"name": "update_settings", "arguments": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}]}, "tags": ["single-call", "nested-json", "numeric-arguments", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-no-call expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for glm/glm45-no-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content: expected 'A simple fact; no tool is needed.', got '\n<think>A simple fact; no tool is needed.'
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Paris is the capital of France.', got '\nParis is the capital of France.'; reasoning_content (whitespace only): expected 'A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>A simple fact; no tool is needed.', got '\nA simple fact; no tool is needed.'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-no-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Paris is the capital of France.",
-  "reasoning_content": "A simple fact; no tool is needed.",
+  "content": "\nParis is the capital of France.",
+  "reasoning_content": "\n<think>A simple fact; no tool is needed.",
   "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Paris is the capital of France.",
-  "reasoning_content": "A simple fact; no tool is needed.",
+  "content": "\nParis is the capital of France.",
+  "reasoning_content": "\nA simple fact; no tool is needed.",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: no-call, reasoning, x-glm45.

Raw output

<think>A simple fact; no tool is needed.</think>
Paris is the capital of France.

Expected parse

{
  "content": "Paris is the capital of France.",
  "reasoning_content": "A simple fact; no tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-no-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>A simple fact; no tool is needed.</think>\nParis is the capital of France.", "output_token_ids": [198, 151350, 32, 4285, 2097, 26, 902, 5392, 374, 4362, 13, 151351, 198, 59190, 374, 279, 6722, 315, 9621, 13], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": "Paris is the capital of France.", "reasoning_content": "A simple fact; no tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-ollama-content-after-call expected_match

Checks that did not pass for glm/glm45-ollama-content-after-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`one`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`special`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`token`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'after tool', got None; tool_calls: expected ['test'], got []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm45-ollama-content-after-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": "after tool",
+  "content": null,
   "reasoning_content": "thinking",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "test"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119. Tags: single-call, empty-arguments, reasoning, text-after-call, x-glm45.

Raw output

<think>thinking</think><tool_call>test</tool_call>after tool

Expected parse

{
  "content": "after tool",
  "reasoning_content": "thinking",
  "tool_calls": [
    {
      "arguments": {},
      "name": "test"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-ollama-content-after-call", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119", "revision": "7af393188defd52d370464de0d2064649cab9b41", "license": "MIT", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright (c) Ollama (MIT)"}, "tools": [{"type": "function", "function": {"name": "test", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>thinking</think><tool_call>test</tool_call>after tool", "output_token_ids": [151350, 81583, 151351, 151352, 1944, 151353, 10689, 5392], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "expected": {"content": "after tool", "reasoning_content": "thinking", "tool_calls": [{"name": "test", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning", "text-after-call", "x-glm45"], "notes": "From Ollama's GLM-4.6 parser test ('tool call with content after'). GLM-4.5/4.6 open <think> themselves, so this is a complete completion after the <|assistant|> generation prompt. The tool schema is a stand-in; the test declares no tools."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail glm/glm45-parallel-mixed-nested expected_match, stream_equals_nonstream, split_invariance, no_leakage, parallel_order

Checks that did not pass for glm/glm45-parallel-mixed-nested
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Check the weather, then search for cafés.', got '\n<think>Check the weather, then search for cafés.'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'; tool_calls: expected ['get_weather', 'search_web'], got ['get_weather']
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'; tool_calls: expected ['get_weather', 'search_web'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'; tool_calls: nonstream ['get_weather', 'search_web'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'; tool_calls: nonstream ['get_weather', 'search_web'], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n\n'; reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:2:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>Check the weather, then search for cafés.', got '\nCheck the weather, then search for cafés.'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search_web'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web'] \| rand:2:8 (fail): content (whitespace only): one None, got '\n'; tool_calls: one ['get_weather'], got ['get_weather', 'search_web'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web']
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'search_web'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'search_web'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-parallel-mixed-nested --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "reasoning_content": "\n<think>Check the weather, then search for cafés.",
   "tool_calls": [
     {

Strategies: one, special

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "reasoning_content": "\nCheck the weather, then search for cafés.",
   "tool_calls": [
     {
@@ -9,18 +9,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max_results": 5,
-          "site": "example.com",
-          "tags": [
-            "coffee",
-            "wifi"
-          ]
-        },
-        "query": "best cafés"
-      },
-      "name": "search_web"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "content": "\n\n",
+  "reasoning_content": "\nCheck the weather, then search for cafés.",
   "tool_calls": [
     {

Strategies: rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "reasoning_content": "\nCheck the weather, then search for cafés.",
   "tool_calls": [
     {

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Check the weather, then search for cafés.",
+  "content": "\n",
+  "reasoning_content": "\nCheck the weather, then search for cafés.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, nested-json, reasoning, x-glm45.

Raw output

<think>Check the weather, then search for cafés.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Zürich</arg_value>
<arg_key>unit</arg_key>
<arg_value>celsius</arg_value>
</tool_call>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>best cafés</arg_value>
<arg_key>filters</arg_key>
<arg_value>{"site": "example.com", "max_results": 5, "tags": ["coffee", "wifi"]}</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Check the weather, then search for cafés.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max_results": 5,
          "site": "example.com",
          "tags": [
            "coffee",
            "wifi"
          ]
        },
        "query": "best cafés"
      },
      "name": "search_web"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-parallel-mixed-nested", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Check the weather, then search for cafés.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Zürich</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>celsius</arg_value>\n</tool_call>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>best cafés</arg_value>\n<arg_key>filters</arg_key>\n<arg_value>{\"site\": \"example.com\", \"max_results\": 5, \"tags\": [\"coffee\", \"wifi\"]}</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 3973, 279, 9101, 11, 1221, 2711, 369, 28996, 5397, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 57, 5186, 713, 151359, 198, 151356, 3843, 151357, 198, 151358, 66, 40026, 151359, 198, 151353, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 15850, 28996, 5397, 151359, 198, 151356, 24533, 151357, 198, 151358, 4913, 9509, 788, 330, 8686, 905, 497, 330, 2810, 13567, 788, 220, 20, 11, 330, 14073, 788, 4383, 77489, 497, 330, 66791, 91669, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Check the weather, then search for cafés.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "celsius"}}, {"name": "search_web", "arguments": {"query": "best cafés", "filters": {"site": "example.com", "max_results": 5, "tags": ["coffee", "wifi"]}}}]}, "tags": ["parallel-calls", "nested-json", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-parallel-same-tool expected_match, stream_equals_nonstream, split_invariance, no_leakage, parallel_order

Checks that did not pass for glm/glm45-parallel-same-tool
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Two cities, so two independent weather lookups.', got '\n<think>Two cities, so two independent weather lookups.'
`expected_match`	`one`	fail	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'; tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'; tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'; reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'; tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'; tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n\n'; reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:2:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>Two cities, so two independent weather lookups.', got '\nTwo cities, so two independent weather lookups.'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:2:8 (fail): content (whitespace only): one None, got '\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather']
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-parallel-same-tool --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "reasoning_content": "\n<think>Two cities, so two independent weather lookups.",
   "tool_calls": [
     {

Strategies: one, special

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "reasoning_content": "\nTwo cities, so two independent weather lookups.",
   "tool_calls": [
     {
@@ -8,10 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "content": "\n\n",
+  "reasoning_content": "\nTwo cities, so two independent weather lookups.",
   "tool_calls": [
     {

Strategies: rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "reasoning_content": "\nTwo cities, so two independent weather lookups.",
   "tool_calls": [
     {

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Two cities, so two independent weather lookups.",
+  "content": "\n",
+  "reasoning_content": "\nTwo cities, so two independent weather lookups.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: parallel-calls, reasoning, x-glm45.

Raw output

<think>Two cities, so two independent weather lookups.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Paris</arg_value>
</tool_call>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Tokyo</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-parallel-same-tool", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Two cities, so two independent weather lookups.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Paris</arg_value>\n</tool_call>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Tokyo</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 11608, 9716, 11, 773, 1378, 9485, 9101, 1401, 8601, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 59190, 151359, 198, 151353, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 52504, 16019, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Two cities, so two independent weather lookups.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["parallel-calls", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-schema-coercion expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-schema-coercion
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'Three second timer labelled 3.', got '\n<think>Three second timer labelled 3.'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`rand:2:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`rand:4:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`stream_equals_nonstream`	`rand:5:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>Three second timer labelled 3.', got '\nThree second timer labelled 3.'
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:2:8 (soft_pass): content (whitespace only): one None, got '\n' \| rand:4:8 (soft_pass): content (whitespace only): one None, got '\n' \| rand:5:8 (soft_pass): content (whitespace only): one None, got '\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-schema-coercion --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Three second timer labelled 3.",
+  "reasoning_content": "\n<think>Three second timer labelled 3.",
   "tool_calls": [
     {

Strategies: one, special, rand:1:8, rand:3:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Three second timer labelled 3.",
+  "reasoning_content": "\nThree second timer labelled 3.",
   "tool_calls": [
     {

Strategies: token, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Three second timer labelled 3.",
+  "content": "\n",
+  "reasoning_content": "\nThree second timer labelled 3.",
   "tool_calls": [
     {

Fixture

Raw output

<think>Three second timer labelled 3.</think>
<tool_call>set_timer
<arg_key>seconds</arg_key>
<arg_value>3</arg_value>
<arg_key>label</arg_key>
<arg_value>3</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Three second timer labelled 3.",
  "tool_calls": [
    {
      "arguments": {
        "label": "3",
        "seconds": 3
      },
      "name": "set_timer"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-schema-coercion", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Three second timer labelled 3.</think>\n<tool_call>set_timer\n<arg_key>seconds</arg_key>\n<arg_value>3</arg_value>\n<arg_key>label</arg_key>\n<arg_value>3</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 19604, 2086, 9018, 59817, 220, 18, 13, 151351, 198, 151352, 746, 16241, 198, 151356, 17381, 151357, 198, 151358, 18, 151359, 198, 151356, 1502, 151357, 198, 151358, 18, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Three second timer labelled 3.", "tool_calls": [{"name": "set_timer", "arguments": {"seconds": 3, "label": "3"}}]}, "tags": ["single-call", "numeric-arguments", "x-schema-coercion", "reasoning", "x-glm45"], "notes": "Both values render as the bare text 3; only the tool schema tells the integer from the string."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-single-call expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected 'The user wants the current weather in Paris, in Celsius.', got '\n<think>The user wants the current weather in Paris, in Celsius.'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:1:8`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`stream_equals_nonstream`	`rand:5:8`	fail	reasoning_content: nonstream '\n<think>The user wants the current weather in Paris, in Celsius.', got '\nThe user wants the current weather in Paris, in Celsius.'
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
+  "reasoning_content": "\n<think>The user wants the current weather in Paris, in Celsius.",
   "tool_calls": [
     {

Strategies: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
+  "reasoning_content": "\nThe user wants the current weather in Paris, in Celsius.",
   "tool_calls": [
     {

Strategies: token

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
+  "content": "\n",
+  "reasoning_content": "\nThe user wants the current weather in Paris, in Celsius.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, reasoning, x-glm45.

Raw output

<think>The user wants the current weather in Paris, in Celsius.</think>
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Paris</arg_value>
<arg_key>unit</arg_key>
<arg_value>celsius</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris, in Celsius.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-single-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>The user wants the current weather in Paris, in Celsius.</think>\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Paris</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>celsius</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 785, 1196, 6801, 279, 1482, 9101, 304, 12089, 11, 304, 60916, 13, 151351, 198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 59190, 151359, 198, 151356, 3843, 151357, 198, 151358, 66, 40026, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris, in Celsius.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "celsius"}}]}, "tags": ["single-call", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-truncated-after-arg-key expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema

Checks that did not pass for glm/glm45-truncated-after-arg-key
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content (whitespace only): nonstream None, got '\n'; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream [], got ['get_weather']
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:2:8 (soft_pass): content (whitespace only): one None, got '\n'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm45-truncated-after-arg-key --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": "
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:2:8

{
  "content": "\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": "
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: truncated, x-glm45.

Expected graceful failure: max_tokens hit between two arguments of an open call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
<arg_key>unit</arg_key>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-truncated-after-arg-key", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n<arg_key>unit</arg_key>\n", "output_token_ids": [198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151356, 3843, 151357, 198], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit between two arguments of an open call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm45"], "notes": "Token prefix of glm/glm45-thinking-disabled."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      198,
      151350,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-unicode-emoji expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for glm/glm45-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	reasoning_content: expected "Search with the user's exact wording.", got "\n<think>Search with the user's exact wording."
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n'; reasoning_content (whitespace only): expected "Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`one`	fail	reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`special`	fail	reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`token`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`rand:1:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`rand:2:8`	fail	reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`rand:3:8`	fail	reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`rand:4:8`	fail	reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`stream_equals_nonstream`	`rand:5:8`	fail	content (whitespace only): nonstream None, got '\n'; reasoning_content: nonstream "\n<think>Search with the user's exact wording.", got "\nSearch with the user's exact wording."
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:1:8 (soft_pass): content (whitespace only): one None, got '\n' \| rand:5:8 (soft_pass): content (whitespace only): one None, got '\n'
`no_leakage`	`nonstream`	fail	reasoning_content contains '<think>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-unicode-emoji --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Search with the user's exact wording.",
+  "reasoning_content": "\n<think>Search with the user's exact wording.",
   "tool_calls": [
     {

Strategies: one, special, rand:2:8, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Search with the user's exact wording.",
+  "reasoning_content": "\nSearch with the user's exact wording.",
   "tool_calls": [
     {

Strategies: token, rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "Search with the user's exact wording.",
+  "content": "\n",
+  "reasoning_content": "\nSearch with the user's exact wording.",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, unicode, reasoning, x-glm45.

Raw output

<think>Search with the user's exact wording.</think>
<tool_call>search_web
<arg_key>query</arg_key>
<arg_value>Zürich café ☕ 東京の天気 🌸</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Search with the user's exact wording.",
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-unicode-emoji", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Search with the user's exact wording.</think>\n<tool_call>search_web\n<arg_key>query</arg_key>\n<arg_value>Zürich café ☕ 東京の天気 🌸</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 5890, 448, 279, 1196, 594, 4734, 59808, 13, 151351, 198, 151352, 1836, 25879, 198, 151356, 1631, 151357, 198, 151358, 57, 5186, 713, 51609, 25048, 243, 60173, 109, 98971, 15755, 98411, 128426, 11157, 234, 116, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Search with the user's exact wording.", "tool_calls": [{"name": "search_web", "arguments": {"query": "Zürich café ☕ 東京の天気 🌸"}}]}, "tags": ["single-call", "unicode", "reasoning", "x-glm45"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm45-vllm-newline-format-content-before expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for glm/glm45-vllm-newline-format-content-before
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	content: expected "I'll check it.", got None
`expected_match`	`special`	fail	content: expected "I'll check it.", got None
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`rand:3:8`	fail	content: expected "I'll check it.", got "I'll check it"
`expected_match`	`rand:4:8`	fail	content: expected "I'll check it.", got "I'll check it"
`stream_equals_nonstream`	`one`	fail	content: nonstream "I'll check it.", got None
`stream_equals_nonstream`	`special`	fail	content: nonstream "I'll check it.", got None
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream "I'll check it.", got "I'll check it. "
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream "I'll check it.", got "I'll check it"
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream "I'll check it.", got "I'll check it"
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got "I'll check it. " \| rand:1:8 (fail): content: one None, got "I'll check it." \| rand:2:8 (fail): content: one None, got "I'll check it." \| rand:3:8 (fail): content: one None, got "I'll check it" \| rand:4:8 (fail): content: one None, got "I'll check it" \| rand:5:8 (fail): content: one None, got "I'll check it."

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-newline-format-content-before --strategy one --strategy rand:3:8 --strategy rand:4:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll check it.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll check it.",
+  "content": "I'll check it",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream, rand:1:8, rand:2:8, rand:5:8

Matches the expected parse.

{
  "content": "I'll check it.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll check it.",
+  "content": "I'll check it. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137. Tags: single-call, text-before-call, x-whitespace-variant-45, x-glm45.

Raw output

I'll check it. <tool_call>get_current_weather
<arg_key>city</arg_key>
<arg_value>Dallas</arg_value>
<arg_key>state</arg_key>
<arg_value>TX</arg_value>
<arg_key>unit</arg_key>
<arg_value>fahrenheit</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll check it.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-newline-format-content-before", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "I'll check it. <tool_call>get_current_weather\n<arg_key>city</arg_key>\n<arg_value>Dallas</arg_value>\n<arg_key>state</arg_key>\n<arg_value>TX</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>fahrenheit</arg_value>\n</tool_call>", "output_token_ids": [40, 3278, 1779, 432, 13, 220, 151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 198, 151358, 85796, 151359, 198, 151356, 2454, 151357, 198, 151358, 22806, 151359, 198, 151356, 3843, 151357, 198, 151358, 69, 47609, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": "I'll check it.", "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}}]}, "tags": ["single-call", "text-before-call", "x-whitespace-variant-45", "x-glm45"], "notes": "vLLM expects the separator space before <tool_call> to be dropped from content."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      198,
      151350,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 3,
  "version": "0.5.20"
}

fail glm/glm45-vllm-newline-format-two-calls expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for glm/glm45-vllm-newline-format-two-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_current_weather', 'get_current_weather'], got ['get_current_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_current_weather', 'get_current_weather'], got ['get_current_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_current_weather', 'get_current_weather'], got ['get_current_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_current_weather', 'get_current_weather'], got ['get_current_weather']
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n'; tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather'] \| rand:1:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather'] \| rand:2:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather'] \| rand:3:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather'] \| rand:4:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather'] \| rand:5:8 (fail): tool_calls: one ['get_current_weather'], got ['get_current_weather', 'get_current_weather']
`parallel_order`	`one`	fail	expected 2 calls ['get_current_weather', 'get_current_weather'], got 1 ['get_current_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_current_weather', 'get_current_weather'], got 1 ['get_current_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-newline-format-two-calls --strategy one --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -8,10 +8,4 @@
       },
       "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Orlando"
-      },
-      "name": "get_current_weather"
     }
   ]

Strategies: nonstream, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Orlando"
      },
      "name": "get_current_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L140-L159. Tags: parallel-calls, x-whitespace-variant-45, x-glm45.

Raw output

<tool_call>get_current_weather
<arg_key>city</arg_key><arg_value>Dallas</arg_value>
</tool_call>
<tool_call>get_current_weather
<arg_key>city</arg_key><arg_value>Orlando</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Orlando"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-newline-format-two-calls", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L140-L159", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "<tool_call>get_current_weather\n<arg_key>city</arg_key><arg_value>Dallas</arg_value>\n</tool_call>\n<tool_call>get_current_weather\n<arg_key>city</arg_key><arg_value>Orlando</arg_value>\n</tool_call>", "output_token_ids": [151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 151358, 85796, 151359, 198, 151353, 198, 151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 151358, 2195, 23627, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas"}}, {"name": "get_current_weather", "arguments": {"city": "Orlando"}}]}, "tags": ["parallel-calls", "x-whitespace-variant-45", "x-glm45"], "notes": "The newline between the two calls is a separator, not content."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      198,
      151350,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 3,
  "version": "0.5.20"
}

fail glm/glm47-bug-three-calls-same-tool expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for glm/glm47-bug-three-calls-same-tool
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n\n'
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got '\n\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n\n'
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got '\n\n'
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got '\n\n'
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got '\n\n'
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got '\n\n'
`split_invariance`	`*`	fail	vs one: token (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:1:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:2:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:3:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:4:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather'] \| rand:5:8 (fail): content (whitespace only): one None, got '\n\n'; tool_calls: one ['get_weather'], got ['get_weather', 'get_weather', 'get_weather']
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm47-bug-three-calls-same-tool --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -8,16 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "B"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "C"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "A"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "B"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "C"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/33324. Tags: parallel-calls, regression, x-call-index-vs-tool-index, x-glm47.

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>A</arg_value></tool_call>
<tool_call>get_weather<arg_key>city</arg_key><arg_value>B</arg_value></tool_call>
<tool_call>get_weather<arg_key>city</arg_key><arg_value>C</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "A"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "B"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "C"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-bug-three-calls-same-tool", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/33324", "revision": "issue opened 2026-08-03", "license": "NOASSERTION", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Quoted from the issue's concrete example (tools [get_weather, search])."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}, {"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>A</arg_value></tool_call>\n<tool_call>get_weather<arg_key>city</arg_key><arg_value>B</arg_value></tool_call>\n<tool_call>get_weather<arg_key>city</arg_key><arg_value>C</arg_value></tool_call>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 32, 151359, 151353, 198, 151352, 455, 68852, 151356, 8923, 151357, 151358, 33, 151359, 151353, 198, 151352, 455, 68852, 151356, 8923, 151357, 151358, 34, 151359, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "A"}}, {"name": "get_weather", "arguments": {"city": "B"}}, {"name": "get_weather", "arguments": {"city": "C"}}]}, "tags": ["parallel-calls", "regression", "x-call-index-vs-tool-index", "x-glm47"], "notes": "SGLang's non-streaming path indexed calls by tool position ([0,0,0]), so OpenAI clients merged them. Tool schemas are minimal stand-ins; the issue only names the tools."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail glm/glm47-call-order-vs-tool-order expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for glm/glm47-call-order-vs-tool-order
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'search_web', 'get_weather'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'search_web', 'get_weather']
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'search_web', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 3 calls ['get_weather', 'search_web', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm47-render.jsonl --id glm/glm47-call-order-vs-tool-order --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/glm/glm47-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -8,16 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "query": "Oslo museums"
-      },
-      "name": "search_web"
-    },
-    {
-      "arguments": {
-        "city": "Bergen"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Weather, then search, then weather again.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "query": "Oslo museums"
      },
      "name": "search_web"
    },
    {
      "arguments": {
        "city": "Bergen"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: parallel-calls, reasoning, x-call-index-vs-tool-index, x-glm47.

Raw output

Weather, then search, then weather again.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Oslo</arg_value></tool_call><tool_call>search_web<arg_key>query</arg_key><arg_value>Oslo museums</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Bergen</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Weather, then search, then weather again.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Oslo"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "query": "Oslo museums"
      },
      "name": "search_web"
    },
    {
      "arguments": {
        "city": "Bergen"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-call-order-vs-tool-order", "family": "glm", "models": ["zai-org/GLM-4.7", "zai-org/GLM-4.7-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "Weather, then search, then weather again.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Oslo</arg_value></tool_call><tool_call>search_web<arg_key>query</arg_key><arg_value>Oslo museums</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Bergen</arg_value></tool_call>", "output_token_ids": [28876, 11, 1221, 2711, 11, 1221, 9101, 1549, 13, 151351, 151352, 455, 68852, 151356, 8923, 151357, 151358, 28764, 385, 151359, 151353, 151352, 1836, 25879, 151356, 1631, 151357, 151358, 28764, 385, 50245, 151359, 151353, 151352, 455, 68852, 151356, 8923, 151357, 151358, 33, 2375, 268, 151359, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Weather, then search, then weather again.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Oslo"}}, {"name": "search_web", "arguments": {"query": "Oslo museums"}}, {"name": "get_weather", "arguments": {"city": "Bergen"}}]}, "tags": ["parallel-calls", "reasoning", "x-call-index-vs-tool-index", "x-glm47"], "notes": "The same tool is called twice around another one; call indices must be 0,1,2, not the tools-list position (https://github.com/sgl-project/sglang/issues/33324)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151350
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value></tool_call>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md"}'
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`token`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:1:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:2:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:3:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:4:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`expected_match`	`rand:5:8`	fail	content: expected None, got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: expected '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`one`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got None; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`special`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got None; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`token`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '; keys go in <arg_key>.</arg_value></tool_call>', got '; keys go in <arg_key>.</arg_value>'; tool_calls[0].arguments: nonstream '{"path": "docs/format.md"}', got '{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}'
`split_invariance`	`*`	fail	vs one: token (fail): content: one None, got '; keys go in <arg_key>.</arg_value>' \| rand:1:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>' \| rand:2:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>' \| rand:3:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>' \| rand:4:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>' \| rand:5:8 (fail): content: one None, got '; keys go in <arg_key>.</arg_value>'
`no_leakage`	`nonstream`	fail	content contains '</tool_call>'; content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`token`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:1:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:2:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:3:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:4:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`no_leakage`	`rand:5:8`	fail	content contains '<arg_key>'; content contains '</arg_value>'
`arguments_json`	`one`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`special`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`token`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:1:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:2:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:3:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:4:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_json`	`rand:5:8`	fail	[0] write_file: arguments are not valid JSON text ('{"path": "docs/format.md", "content": "Wrap each call in <tool_call> and}': Unterminated string starting at: line 1 column 39 (char 38))
`arguments_schema`	`nonstream`	fail	[0] write_file: $: 'content' is a required property
`arguments_schema`	`one`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] write_file: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm47-render.jsonl --id glm/glm47-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/glm/glm47-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,9 +1,8 @@
 {
-  "content": null,
+  "content": "; keys go in <arg_key>.</arg_value></tool_call>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
         "path": "docs/format.md"
       },

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and}"
       },
       "name": "write_file"

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,9 @@
 {
-  "content": null,
+  "content": "; keys go in <arg_key>.</arg_value>",
   "reasoning_content": "Write the format note to the docs file.",
   "tool_calls": [
     {
       "arguments": {
-        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
-        "path": "docs/format.md"
+        "<arguments_raw, not valid JSON>": "{\"path\": \"docs/format.md\", \"content\": \"Wrap each call in <tool_call> and}"
       },
       "name": "write_file"

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: single-call, marker-in-arguments, reasoning, x-glm47.

Raw output

Write the format note to the docs file.</think><tool_call>write_file<arg_key>path</arg_key><arg_value>docs/format.md</arg_value><arg_key>content</arg_key><arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Write the format note to the docs file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.",
        "path": "docs/format.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-marker-in-arguments", "family": "glm", "models": ["zai-org/GLM-4.7", "zai-org/GLM-4.7-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "Write the format note to the docs file.</think><tool_call>write_file<arg_key>path</arg_key><arg_value>docs/format.md</arg_value><arg_key>content</arg_key><arg_value>Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>.</arg_value></tool_call>", "output_token_ids": [7984, 279, 3561, 5185, 311, 279, 26257, 1034, 13, 151351, 151352, 4934, 2458, 151356, 2343, 151357, 151358, 14111, 91564, 21274, 151359, 151356, 1796, 151357, 151358, 26701, 1817, 1618, 304, 220, 151352, 323, 220, 151353, 26, 6894, 728, 304, 220, 151356, 13, 151359, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Write the format note to the docs file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/format.md", "content": "Wrap each call in <tool_call> and </tool_call>; keys go in <arg_key>."}}]}, "tags": ["single-call", "marker-in-arguments", "reasoning", "x-glm47"], "notes": "The string value contains <tool_call>, </tool_call> and <arg_key> (tokenized as the marker tokens). It does not contain </arg_value>, so the value is still delimited unambiguously."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151350
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-missing-last-close-arg-value expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-missing-last-close-arg-value
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin"}'
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"city": "Berlin"}', got '{"city": "Berlin", "unit": "celsius}'
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius}': Unterminated string starting at: line 1 column 28 (char 27))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-missing-last-close-arg-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/glm/truncated-malformed.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Berlin",
-        "unit": "celsius"
+        "city": "Berlin"
       },
       "name": "get_weather"

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Berlin",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": \"celsius}"
       },
       "name": "get_weather"

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/57826. Tags: single-call, malformed, regression, x-glm47, x-missing-close-arg, x-derived.

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-missing-last-close-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/57826", "revision": "issue opened 2026-09-20", "license": "NOASSERTION", "generator": "scripts/fixtures/glm/build.py", "attribution": "Issue text: \"a tool call that omits the last closing arg_value tag yields non-streaming arguments without the trailing parameter.\""}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin", "unit": "celsius"}}]}, "tags": ["single-call", "malformed", "regression", "x-glm47", "x-missing-close-arg", "x-derived"], "notes": "raw_output is NOT quoted from the issue (it gives no GLM string): it is derived from glm/glm47-thinking-disabled by deleting the last </arg_value> token, the malformation #57826 describes. </tool_call> still closes the call, so the trailing value is recoverable; the issue (and the qwen3 fix it mirrors, vLLM #57707) expects the parser to keep it."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-parallel-same-tool expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for glm/glm47-parallel-same-tool
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather']
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm47-render.jsonl --id glm/glm47-parallel-same-tool --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/glm/glm47-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -8,10 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: parallel-calls, reasoning, x-glm47.

Raw output

Two cities, so two independent weather lookups.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Tokyo</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-parallel-same-tool", "family": "glm", "models": ["zai-org/GLM-4.7", "zai-org/GLM-4.7-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "Two cities, so two independent weather lookups.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Tokyo</arg_value></tool_call>", "output_token_ids": [11608, 9716, 11, 773, 1378, 9485, 9101, 1401, 8601, 13, 151351, 151352, 455, 68852, 151356, 8923, 151357, 151358, 59190, 151359, 151353, 151352, 455, 68852, 151356, 8923, 151357, 151358, 52504, 16019, 151359, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Two cities, so two independent weather lookups.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["parallel-calls", "reasoning", "x-glm47"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151350
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-truncated-before-close-call expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-truncated-before-close-call
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": "celsius"': Expecting ',' delimiter: line 1 column 37 (char 36))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-before-close-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": \"celsius\""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit after the last </arg_value>; </tool_call> is missing. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-before-close-call", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151359], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit after the last </arg_value>; </tool_call> is missing.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled. A parser should not emit a call that never closed."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-truncated-inside-arg-value expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-truncated-inside-arg-value
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>', got None; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('{"city": "Berlin", "unit": ': Expecting value: line 1 column 28 (char 27))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-inside-arg-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Berlin\", \"unit\": "
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit inside the second <arg_value>; the call never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-inside-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit inside the second <arg_value>; the call never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

fail glm/glm47-vllm-args-with-newlines expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm47-vllm-args-with-newlines
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm47-vllm-args-with-newlines --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Beijing"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L154-L158. Tags: single-call, x-whitespace-variant-45, x-glm47.

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Beijing</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-vllm-args-with-newlines", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L154-L158", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_date", "parameters": {}}}, {"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "date": {"type": "string"}}}}}], "raw_output": "<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Beijing</arg_value>\n</tool_call>", "output_token_ids": [151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 3430, 23584, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Beijing"}}]}, "tags": ["single-call", "x-whitespace-variant-45", "x-glm47"], "notes": "GLM-4.5-style newlines inside a GLM-4.7 completion; the glm47 parser must accept both variants."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail glm/glm53-parallel-same-tool expected_match, stream_equals_nonstream, split_invariance, parallel_order

Checks that did not pass for glm/glm53-parallel-same-tool
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'get_weather'], got ['get_weather']
`split_invariance`	`*`	fail	vs one: token (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:1:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:2:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:3:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:4:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather'] \| rand:5:8 (fail): tool_calls: one ['get_weather'], got ['get_weather', 'get_weather']
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'get_weather'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm53-render.jsonl --id glm/glm53-parallel-same-tool --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/glm/glm53-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -8,10 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
     }
   ]

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-5.3/blob/aca966e4e02791568aa6a4ced368624b3d897f42/chat_template.jinja. Tags: parallel-calls, reasoning, x-glm53.

Raw output

Two cities, so two independent weather lookups.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Tokyo</arg_value></tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": "Two cities, so two independent weather lookups.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm53-parallel-same-tool", "family": "glm", "models": ["zai-org/GLM-5.3"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-5.3/blob/aca966e4e02791568aa6a4ced368624b3d897f42/chat_template.jinja", "revision": "aca966e4e02791568aa6a4ced368624b3d897f42", "license": "LicenseRef-glm-5.3", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "3740abcea51c45830cb3ca562084ad5fb2ef53589376f73332e9886f93ade41c"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "Two cities, so two independent weather lookups.</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call><tool_call>get_weather<arg_key>city</arg_key><arg_value>Tokyo</arg_value></tool_call>", "output_token_ids": [11608, 9716, 11, 773, 1378, 9485, 9101, 1401, 8601, 13, 154842, 154843, 455, 68852, 154847, 8923, 154848, 154849, 59190, 154850, 154844, 154843, 455, 68852, 154847, 8923, 154848, 154849, 52504, 16019, 154850, 154844], "tokenizer": {"repo": "zai-org/GLM-5.3", "revision": "aca966e4e02791568aa6a4ced368624b3d897f42", "mode": "hf"}, "thinking": true, "expected": {"content": null, "reasoning_content": "Two cities, so two independent weather lookups.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["parallel-calls", "reasoning", "x-glm53"]}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "3740abcea51c45830cb3ca562084ad5fb2ef53589376f73332e9886f93ade41c",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GlmMoeDsaForCausalLM"
    ],
    "model_type": "glm_moe_dsa"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-5.3",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      154828,
      154841
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 154820,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-5.3",
    "revision": "aca966e4e02791568aa6a4ced368624b3d897f42",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

soft pass glm/glm45-thinking-disabled expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for glm/glm45-thinking-disabled
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got '\n'
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got '\n'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got '\n'
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got '\n'
`split_invariance`	`*`	soft pass	vs one: token (soft_pass): content (whitespace only): one None, got '\n' \| rand:2:8 (soft_pass): content (whitespace only): one None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-thinking-disabled --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, x-glm45.

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
<arg_key>unit</arg_key>
<arg_value>celsius</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-thinking-disabled", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>celsius</arg_value>\n</tool_call>", "output_token_ids": [198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151356, 3843, 151357, 198, 151358, 66, 40026, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin", "unit": "celsius"}}]}, "tags": ["single-call", "x-glm45"], "notes": "enable_thinking=false: the generation prompt pre-fills the (empty) reasoning."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm45"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.5",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      198,
      151350,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.5",
    "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm4MoeDetector",
  "tool_call_parser": "glm45",
  "tool_choice": "auto",
  "tools_offered": 9,
  "version": "0.5.20"
}

soft pass glm/glm47-vllm-whitespace-only-content expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm47-vllm-whitespace-only-content
Check	Strategy	Result	Detail
`expected_match`	`one`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`special`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`token`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected None, got ' \n '
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected None, got ' \n '
`stream_equals_nonstream`	`one`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`special`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`rand:1:8`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`rand:2:8`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`rand:3:8`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`rand:4:8`	soft pass	content (whitespace only): nonstream None, got ' \n '
`stream_equals_nonstream`	`rand:5:8`	soft pass	content (whitespace only): nonstream None, got ' \n '

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/glm/imported.jsonl --id glm/glm47-vllm-whitespace-only-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_current_date"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "  \n  ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L185-L188. Tags: single-call, empty-arguments, x-whitespace-variant-47, x-glm47.

Raw output

  
  <tool_call>get_current_date</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_current_date"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-vllm-whitespace-only-content", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L185-L188", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_date", "parameters": {}}}, {"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "date": {"type": "string"}}}}}], "raw_output": "  \n  <tool_call>get_current_date</tool_call>", "output_token_ids": [2303, 256, 151352, 455, 11075, 4164, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_date", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-whitespace-variant-47", "x-glm47"], "notes": "Whitespace-only text before the call is not content (vLLM asserts content is None)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "glm45",
    "tool_call_parser": "glm47"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Glm4MoeForCausalLM"
    ],
    "model_type": "glm4_moe"
  },
  "hf_config_error": null,
  "model": "zai-org/GLM-4.7",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      151337,
      151351
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Glm45Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "glm45",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 151329,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|endoftext|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "zai-org/GLM-4.7",
    "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Glm47MoeDetector",
  "tool_call_parser": "glm47",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model