GLM (4.5, 4.6, 4.7, 5.x) on llamacpp a25c9865

fail 80% strict pass 41 pass · 2 soft pass · 8 fail

Run

Engine

llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)

Run

2026-09-26T05:36:17Z to 2026-09-26T05:37:20Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}

Results file

data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	41	2	5	85%
`expected_error`	0	0	3	0%
`stream_equals_nonstream`	48	0	3	94%
`no_leakage`	50	0	1	98%
`arguments_json`	37	0	6	86%
`arguments_schema`	37	0	6	86%
`parallel_order`	7	0	1	88%

8 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail glm/glm45-ollama-content-after-call expected_match, stream_equals_nonstream

Checks that did not pass for glm/glm45-ollama-content-after-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream None, got 'thinking'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/imported.jsonl --id glm/glm45-ollama-content-after-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
-  "content": "after tool",
-  "reasoning_content": "thinking",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "test"
-    }
-  ]
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
-  "content": "after tool",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": "thinking",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "test"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119. Tags: single-call, empty-arguments, reasoning, text-after-call, x-glm45.

Raw output

<think>thinking</think><tool_call>test</tool_call>after tool

Expected parse

{
  "content": "after tool",
  "reasoning_content": "thinking",
  "tool_calls": [
    {
      "arguments": {},
      "name": "test"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-ollama-content-after-call", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ollama/ollama/blob/7af393188defd52d370464de0d2064649cab9b41/model/parsers/glm46_test.go#L108-L119", "revision": "7af393188defd52d370464de0d2064649cab9b41", "license": "MIT", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright (c) Ollama (MIT)"}, "tools": [{"type": "function", "function": {"name": "test", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<think>thinking</think><tool_call>test</tool_call>after tool", "output_token_ids": [151350, 81583, 151351, 151352, 1944, 151353, 10689, 5392], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "expected": {"content": "after tool", "reasoning_content": "thinking", "tool_calls": [{"name": "test", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning", "text-after-call", "x-glm45"], "notes": "From Ollama's GLM-4.6 parser test ('tool call with content after'). GLM-4.5/4.6 open <think> themselves, so this is a complete completion after the <|assistant|> generation prompt. The tool schema is a stand-in; the test declares no tools."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2a0d3c7b417677c267dd391b5fe8934447c0852da9f1470987c0ad379c526d41",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

fail glm/glm45-truncated-after-arg-key expected_error, arguments_json, arguments_schema

Checks that did not pass for glm/glm45-truncated-after-arg-key
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit between two arguments of an open call.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":': Expecting value: line 1 column 25 (char 24))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":': Expecting value: line 1 column 25 (char 24))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm45-truncated-after-arg-key --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":\"Berlin\",\"unit\":"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: truncated, x-glm45.

Expected graceful failure: max_tokens hit between two arguments of an open call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
<arg_key>unit</arg_key>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-truncated-after-arg-key", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n<arg_key>unit</arg_key>\n", "output_token_ids": [198, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151356, 3843, 151357, 198], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit between two arguments of an open call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm45"], "notes": "Token prefix of glm/glm45-thinking-disabled."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>\n<think></think>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cb462282ca1bcf3b800eb4ca1f3c8fd0d5381a279f1d124a7f65e7435e64896c",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

fail glm/glm45-vllm-coerces-schema-types expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for glm/glm45-vllm-coerces-schema-types
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['calculate']
`arguments_json`	`token`	fail	[0] calculate: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] calculate: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-coerces-schema-types --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,15 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "a": 42,
-        "b": 3.14,
-        "enabled": true,
-        "operation": "add"
-      },
-      "name": "calculate"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "a": 42,
-        "b": 3.14,
-        "enabled": true,
-        "operation": "add"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "calculate"

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L162-L180. Tags: single-call, numeric-arguments, x-schema-coercion, x-glm45.

Raw output

<tool_call>calculate
<arg_key>operation</arg_key><arg_value>add</arg_value>
<arg_key>a</arg_key><arg_value>42</arg_value>
<arg_key>b</arg_key><arg_value>3.14</arg_value>
<arg_key>enabled</arg_key><arg_value>true</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "a": 42,
        "b": 3.14,
        "enabled": true,
        "operation": "add"
      },
      "name": "calculate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-coerces-schema-types", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L162-L180", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "<tool_call>calculate\n<arg_key>operation</arg_key><arg_value>add</arg_value>\n<arg_key>a</arg_key><arg_value>42</arg_value>\n<arg_key>b</arg_key><arg_value>3.14</arg_value>\n<arg_key>enabled</arg_key><arg_value>true</arg_value>\n</tool_call>", "output_token_ids": [151352, 35419, 198, 151356, 9259, 151357, 151358, 718, 151359, 198, 151356, 64, 151357, 151358, 101961, 151359, 198, 151356, 65, 151357, 151358, 18, 13, 99367, 151359, 198, 151356, 15856, 151357, 151358, 1866, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"operation": "add", "a": 42, "b": 3.14, "enabled": true}}]}, "tags": ["single-call", "numeric-arguments", "x-schema-coercion", "x-glm45"]}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>\n<think></think>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "551203dc49adf37b9b2a4103f226339a075950211d4555c4a40758a1e6a9cbab",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

fail glm/glm45-vllm-newline-format-two-calls expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for glm/glm45-vllm-newline-format-two-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_current_weather']
`arguments_json`	`token`	fail	[0] get_current_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_current_weather: arguments are not a JSON object; not validated
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_current_weather', 'get_current_weather'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_current_weather', 'get_current_weather'], got 1 ['get_current_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:5:8 fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-newline-format-two-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, rand:4:8

--- expected
+++ observed
@@ -1,18 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Dallas"
-      },
-      "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Orlando"
-      },
-      "name": "get_current_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,15 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Dallas"
-      },
-      "name": "get_current_weather"
-    },
-    {
-      "arguments": {
-        "city": "Orlando"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_current_weather"

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L140-L159. Tags: parallel-calls, x-whitespace-variant-45, x-glm45.

Raw output

<tool_call>get_current_weather
<arg_key>city</arg_key><arg_value>Dallas</arg_value>
</tool_call>
<tool_call>get_current_weather
<arg_key>city</arg_key><arg_value>Orlando</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas"
      },
      "name": "get_current_weather"
    },
    {
      "arguments": {
        "city": "Orlando"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-newline-format-two-calls", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L140-L159", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "<tool_call>get_current_weather\n<arg_key>city</arg_key><arg_value>Dallas</arg_value>\n</tool_call>\n<tool_call>get_current_weather\n<arg_key>city</arg_key><arg_value>Orlando</arg_value>\n</tool_call>", "output_token_ids": [151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 151358, 85796, 151359, 198, 151353, 198, 151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 151358, 2195, 23627, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas"}}, {"name": "get_current_weather", "arguments": {"city": "Orlando"}}]}, "tags": ["parallel-calls", "x-whitespace-variant-45", "x-glm45"], "notes": "The newline between the two calls is a separator, not content."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>\n<think></think>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "551203dc49adf37b9b2a4103f226339a075950211d4555c4a40758a1e6a9cbab",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

fail glm/glm47-missing-last-close-arg-value expected_match, no_leakage, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-missing-last-close-arg-value
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city":"Berlin","unit":"celsius</tool_call>'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"city": "Berlin", "unit": "celsius"}', got '{"city":"Berlin","unit":"celsius</tool_call>'
`no_leakage`	`nonstream`	fail	tool_calls[0].arguments contains '</tool_call>'
`no_leakage`	`token`	fail	tool_calls[0].arguments contains '</tool_call>'
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"celsius</tool_call>': Unterminated string starting at: line 1 column 25 (char 24))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"celsius</tool_call>': Unterminated string starting at: line 1 column 25 (char 24))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-missing-last-close-arg-value --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/glm/truncated-malformed.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Berlin",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\":\"Berlin\",\"unit\":\"celsius</tool_call>"
       },
       "name": "get_weather"

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/57826. Tags: single-call, malformed, regression, x-glm47, x-missing-close-arg, x-derived.

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-missing-last-close-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/57826", "revision": "issue opened 2026-09-20", "license": "NOASSERTION", "generator": "scripts/fixtures/glm/build.py", "attribution": "Issue text: \"a tool call that omits the last closing arg_value tag yields non-streaming arguments without the trailing parameter.\""}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</tool_call>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin", "unit": "celsius"}}]}, "tags": ["single-call", "malformed", "regression", "x-glm47", "x-missing-close-arg", "x-derived"], "notes": "raw_output is NOT quoted from the issue (it gives no GLM string): it is derived from glm/glm47-thinking-disabled by deleting the last </arg_value> token, the malformation #57826 describes. </tool_call> still closes the call, so the trailing value is recoverable; the issue (and the qwen3 fix it mirrors, vLLM #57707) expects the parser to keep it."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|></think>",
  "fixture_tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "format": "peg-native",
  "generation_prompt": "<|assistant|></think>",
  "model": "zai-org/GLM-4.7",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.7"
    }
  },
  "template_identical_to": "models/templates/GLM-4.7-Flash.jinja",
  "template_path": "zai-org--GLM-4.7.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cb462282ca1bcf3b800eb4ca1f3c8fd0d5381a279f1d124a7f65e7435e64896c",
  "vocab_gguf": "zai-org--GLM-4.7.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "6b9f8ab9bceda63102fc1fc9775e79287dd4c921eaefa03a9170ab4a14ae6570"
}

fail glm/glm47-truncated-before-close-call expected_error, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-truncated-before-close-call
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit after the last </arg_value>; </tool_call> is missing.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"celsius"': Expecting ',' delimiter: line 1 column 34 (char 33))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"celsius"': Expecting ',' delimiter: line 1 column 34 (char 33))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-before-close-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":\"Berlin\",\"unit\":\"celsius\""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit after the last </arg_value>; </tool_call> is missing. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-before-close-call", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358, 66, 40026, 151359], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit after the last </arg_value>; </tool_call> is missing.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled. A parser should not emit a call that never closed."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|></think>",
  "fixture_tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "format": "peg-native",
  "generation_prompt": "<|assistant|></think>",
  "model": "zai-org/GLM-4.7",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.7"
    }
  },
  "template_identical_to": "models/templates/GLM-4.7-Flash.jinja",
  "template_path": "zai-org--GLM-4.7.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cb462282ca1bcf3b800eb4ca1f3c8fd0d5381a279f1d124a7f65e7435e64896c",
  "vocab_gguf": "zai-org--GLM-4.7.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "6b9f8ab9bceda63102fc1fc9775e79287dd4c921eaefa03a9170ab4a14ae6570"
}

fail glm/glm47-truncated-inside-arg-value expected_error, arguments_json, arguments_schema

Checks that did not pass for glm/glm47-truncated-inside-arg-value
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for max_tokens hit inside the second <arg_value>; the call never closes.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"': Unterminated string starting at: line 1 column 25 (char 24))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Berlin","unit":"': Unterminated string starting at: line 1 column 25 (char 24))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/truncated-malformed.jsonl --id glm/glm47-truncated-inside-arg-value --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/glm/truncated-malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":\"Berlin\",\"unit\":\""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja. Tags: truncated, x-glm47.

Expected graceful failure: max_tokens hit inside the second <arg_value>; the call never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-truncated-inside-arg-value", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.7/blob/602d01efcdd332c5238ca4bcede555defbe83eb7/chat_template.jinja", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "<tool_call>get_weather<arg_key>city</arg_key><arg_value>Berlin</arg_value><arg_key>unit</arg_key><arg_value>", "output_token_ids": [151352, 455, 68852, 151356, 8923, 151357, 151358, 93615, 151359, 151356, 3843, 151357, 151358], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected_error": {"reason": "max_tokens hit inside the second <arg_value>; the call never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-glm47"], "notes": "Token prefix of glm/glm47-thinking-disabled."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|></think>",
  "fixture_tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "format": "peg-native",
  "generation_prompt": "<|assistant|></think>",
  "model": "zai-org/GLM-4.7",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.7"
    }
  },
  "template_identical_to": "models/templates/GLM-4.7-Flash.jinja",
  "template_path": "zai-org--GLM-4.7.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cb462282ca1bcf3b800eb4ca1f3c8fd0d5381a279f1d124a7f65e7435e64896c",
  "vocab_gguf": "zai-org--GLM-4.7.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "6b9f8ab9bceda63102fc1fc9775e79287dd4c921eaefa03a9170ab4a14ae6570"
}

fail glm/glm47-vllm-args-with-newlines expected_match

Checks that did not pass for glm/glm47-vllm-args-with-newlines
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/imported.jsonl --id glm/glm47-vllm-args-with-newlines --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Beijing"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L154-L158. Tags: single-call, x-whitespace-variant-45, x-glm47.

Raw output

<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Beijing</arg_value>
</tool_call>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm47-vllm-args-with-newlines", "family": "glm", "models": ["zai-org/GLM-4.7"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm47_moe_tool_parser.py#L154-L158", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_date", "parameters": {}}}, {"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "date": {"type": "string"}}}}}], "raw_output": "<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Beijing</arg_value>\n</tool_call>", "output_token_ids": [151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 3430, 23584, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.7", "revision": "602d01efcdd332c5238ca4bcede555defbe83eb7", "mode": "hf"}, "generation_prompt": "<|assistant|></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Beijing"}}]}, "tags": ["single-call", "x-whitespace-variant-45", "x-glm47"], "notes": "GLM-4.5-style newlines inside a GLM-4.7 completion; the glm47 parser must accept both variants."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|></think>",
  "fixture_tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "format": "peg-native",
  "generation_prompt": "<|assistant|></think>",
  "model": "zai-org/GLM-4.7",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.7"
    }
  },
  "template_identical_to": "models/templates/GLM-4.7-Flash.jinja",
  "template_path": "zai-org--GLM-4.7.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "d63ad536c3c81880043e22ec7fd08db42b4d8fb7c89c7138bc562bfa25281375",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.7@602d01efcdd332c5238ca4bcede555defbe83eb7",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "877c7fb0bdb22a84ed125e27b8fb4001f46eac7260e234507922cbed1a646e31",
  "vocab_gguf": "zai-org--GLM-4.7.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "6b9f8ab9bceda63102fc1fc9775e79287dd4c921eaefa03a9170ab4a14ae6570"
}

soft pass glm/glm45-content-before-call expected_match

Checks that did not pass for glm/glm45-content-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "I'll look that up for you.\n"
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll look that up for you.", got "I'll look that up for you.\n"

Synthetic strategies (not counted): expected_match@one soft pass, expected_match@special soft pass, expected_match@rand:1:8 soft pass, expected_match@rand:2:8 soft pass, expected_match@rand:3:8 soft pass, expected_match@rand:4:8 soft pass, expected_match@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/glm45-render.jsonl --id glm/glm45-content-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/glm/glm45-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look that up for you.",
+  "content": "I'll look that up for you.\n",
   "reasoning_content": "Need live data, so call the tool.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja. Tags: single-call, text-before-call, reasoning, x-glm45.

Raw output

<think>Need live data, so call the tool.</think>
I'll look that up for you.
<tool_call>get_weather
<arg_key>city</arg_key>
<arg_value>Berlin</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll look that up for you.",
  "reasoning_content": "Need live data, so call the tool.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Berlin"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-content-before-call", "family": "glm", "models": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air", "zai-org/GLM-4.6"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/zai-org/GLM-4.5/blob/cbb2c7cfb52fa128a9660cb1a7a78e017899e115/chat_template.jinja", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "license": "MIT", "generator": "scripts/fixtures/glm/build.py", "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object", "properties": {"site": {"type": "string"}, "max_results": {"type": "integer"}, "tags": {"type": "array", "items": {"type": "string"}}}}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_timer", "description": "Start a countdown timer.", "parameters": {"type": "object", "properties": {"seconds": {"type": "integer"}, "label": {"type": "string"}}, "required": ["seconds"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by its id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "reference": {"type": "string"}}, "required": ["order_id"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "\n<think>Need live data, so call the tool.</think>\nI'll look that up for you.\n<tool_call>get_weather\n<arg_key>city</arg_key>\n<arg_value>Berlin</arg_value>\n</tool_call>", "output_token_ids": [198, 151350, 23591, 3887, 821, 11, 773, 1618, 279, 5392, 13, 151351, 198, 40, 3278, 1401, 429, 705, 369, 498, 624, 151352, 455, 68852, 198, 151356, 8923, 151357, 198, 151358, 93615, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "thinking": true, "expected": {"content": "I'll look that up for you.", "reasoning_content": "Need live data, so call the tool.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Berlin"}}]}, "tags": ["single-call", "text-before-call", "reasoning", "x-glm45"], "notes": "The template separates content from the call (GLM-4.5 with a newline); the correct content is the message text without that separator."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cb462282ca1bcf3b800eb4ca1f3c8fd0d5381a279f1d124a7f65e7435e64896c",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

soft pass glm/glm45-vllm-newline-format-content-before expected_match

Checks that did not pass for glm/glm45-vllm-newline-format-content-before
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "
`expected_match`	`token`	soft pass	content (whitespace only): expected "I'll check it.", got "I'll check it. "

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/glm/imported.jsonl --id glm/glm45-vllm-newline-format-content-before --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/glm/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll check it.",
+  "content": "I'll check it. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137. Tags: single-call, text-before-call, x-whitespace-variant-45, x-glm45.

Raw output

I'll check it. <tool_call>get_current_weather
<arg_key>city</arg_key>
<arg_value>Dallas</arg_value>
<arg_key>state</arg_key>
<arg_value>TX</arg_value>
<arg_key>unit</arg_key>
<arg_value>fahrenheit</arg_value>
</tool_call>

Expected parse

{
  "content": "I'll check it.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Dallas",
        "state": "TX",
        "unit": "fahrenheit"
      },
      "name": "get_current_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "glm/glm45-vllm-newline-format-content-before", "family": "glm", "models": ["zai-org/GLM-4.5"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_glm4_moe_tool_parser.py#L114-L137", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/glm/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_current_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "state": {"type": "string"}, "unit": {"type": "string"}}}}}, {"type": "function", "function": {"name": "calculate", "parameters": {"type": "object", "properties": {"operation": {"type": "string"}, "a": {"type": "number"}, "b": {"type": "number"}, "enabled": {"type": "boolean"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {}}}], "raw_output": "I'll check it. <tool_call>get_current_weather\n<arg_key>city</arg_key>\n<arg_value>Dallas</arg_value>\n<arg_key>state</arg_key>\n<arg_value>TX</arg_value>\n<arg_key>unit</arg_key>\n<arg_value>fahrenheit</arg_value>\n</tool_call>", "output_token_ids": [40, 3278, 1779, 432, 13, 220, 151352, 455, 11075, 68852, 198, 151356, 8923, 151357, 198, 151358, 85796, 151359, 198, 151356, 2454, 151357, 198, 151358, 22806, 151359, 198, 151356, 3843, 151357, 198, 151358, 69, 47609, 151359, 198, 151353], "tokenizer": {"repo": "zai-org/GLM-4.5", "revision": "cbb2c7cfb52fa128a9660cb1a7a78e017899e115", "mode": "hf"}, "generation_prompt": "<|assistant|>\n<think></think>", "thinking": false, "expected": {"content": "I'll check it.", "reasoning_content": null, "tool_calls": [{"name": "get_current_weather", "arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}}]}, "tags": ["single-call", "text-before-call", "x-whitespace-variant-45", "x-glm45"], "notes": "vLLM expects the separator space before <tool_call> to be dropped from content."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|endoftext|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|assistant|>\n<think></think>",
  "fixture_tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "format": "peg-native",
  "generation_prompt": "<|assistant|>\n<think></think>",
  "model": "zai-org/GLM-4.5",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<think>",
    "</think>",
    "<tool_call>",
    "</tool_call>",
    "<arg_key>",
    "</arg_key>",
    "<arg_value>",
    "</arg_value>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after zai-org/GLM-4.5"
    }
  },
  "template_identical_to": null,
  "template_path": "zai-org--GLM-4.5.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "44f815868bf02fa458dd2f741a338046f4bf45f398eb6d067766726b9d96cce3",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "zai-org/GLM-4.5@cbb2c7cfb52fa128a9660cb1a7a78e017899e115",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "551203dc49adf37b9b2a4103f226339a075950211d4555c4a40758a1e6a9cbab",
  "vocab_gguf": "zai-org--GLM-4.5.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "661727b19e55df5ba0f05b852b3126ae4aefe0d7c239975253f4855e10452df2"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model