CanIToolCall caniuse.com for tool calling

Gemma 4 (<|tool_call>call:NAME{...} object notation) on vllm 0.30.0

fail 62% strict pass 30 pass · 2 soft pass · 16 fail

Run

Engine
vllm 0.30.0
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}
Results file
data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 30 2 12 0 68%
expected_error 0 0 4 0 0%
stream_equals_nonstream 42 2 4 0 88%
split_invariance 48 0 0 0 100%
no_leakage 47 0 1 0 98%
arguments_json 41 0 0 0 100%
arguments_schema 39 0 2 0 95%
parallel_order 4 0 0 0 100%

Fixtures needing attention

fail gemma4/bug-ollama-18390-key-with-spaces expected_match
Checks that did not pass for gemma4/bug-ollama-18390-key-with-spaces
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchone failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchspecial failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchtoken failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}', got '{"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": "0"}]]}}}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/bug-reports.jsonl --id gemma4/bug-ollama-18390-key-with-spaces --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/gemma4/bug-reports.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -10,5 +10,5 @@
               [
                 {
-                  "index": 0,
+                  "index": "0",
                   "node": "X",
                   "type": "main"

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18390. Tags: single-call, nested-json, regression, x-key-with-space.

Raw output

<|tool_call>call:create_workflow{name:<|"|>Demo<|"|>, connections:{Basic LLM Chain:{main:[[{node:<|"|>X<|"|>, type:<|"|>main<|"|>, index:0}]]}}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "connections": {
          "Basic LLM Chain": {
            "main": [
              [
                {
                  "index": 0,
                  "node": "X",
                  "type": "main"
                }
              ]
            ]
          }
        },
        "name": "Demo"
      },
      "name": "create_workflow"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/bug-ollama-18390-key-with-spaces", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18390", "revision": "issue body (2026-09-11)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "create_workflow", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "connections": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:create_workflow{name:<|\"|>Demo<|\"|>, connections:{Basic LLM Chain:{main:[[{node:<|\"|>X<|\"|>, type:<|\"|>main<|\"|>, index:0}]]}}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 3663, 236779, 95778, 236782, 1201, 236787, 52, 33846, 52, 236764, 13792, 29616, 18397, 32050, 236792, 35211, 29616, 3241, 24845, 53103, 5335, 236787, 52, 236917, 52, 236764, 1722, 236787, 52, 3241, 52, 236764, 3546, 236787, 236771, 236783, 10660, 19465, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_workflow", "arguments": {"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}}]}, "tags": ["single-call", "nested-json", "regression", "x-key-with-space"], "notes": "The 'bare key with spaces' input of the issue's parser-level Go test. Ollama dropped the whole call. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/edge-values-key-with-space expected_match
Checks that did not pass for gemma4/edge-values-key-with-space
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchone failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchspecial failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchtoken failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": null, "obj": {"a": ["1", "t"], "B": "1"}, "s": "x,y} \\"q\\""}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/edge.jsonl --id gemma4/edge-values-key-with-space --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,7 +9,7 @@
         "none": null,
         "obj": {
-          "B": 1,
+          "B": "1",
           "a": [
-            1,
+            "1",
             "t"
           ]

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, numeric-arguments, string-escapes, nested-json, x-key-with-space, x-dictsort-order.

Raw output

<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|"|>t<|"|>],B:1},s:<|"|>x,y} "q"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "flag": false,
        "key with space": 1.5e-05,
        "none": null,
        "obj": {
          "B": 1,
          "a": [
            1,
            "t"
          ]
        },
        "s": "x,y} \"q\""
      },
      "name": "record"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/edge-values-key-with-space", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "record", "description": "Store a record with arbitrary keys.", "parameters": {"type": "object", "properties": {"key with space": {"type": "number"}, "s": {"type": "string"}, "flag": {"type": "boolean"}, "none": {"anyOf": [{"type": "string"}, {"type": "null"}], "nullable": true}, "obj": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|\"|>t<|\"|>],B:1},s:<|\"|>x,y} \"q\"<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17789, 236782, 16213, 236787, 4530, 236764, 2478, 607, 2557, 236787, 236770, 236761, 236810, 236744, 236772, 236771, 236810, 236764, 9745, 236787, 6455, 236764, 7837, 29616, 236746, 24845, 236770, 236764, 52, 236745, 52, 1604, 236799, 236787, 236770, 1263, 236751, 236787, 52, 236781, 236764, 236762, 236783, 623, 236809, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "record", "arguments": {"key with space": 1.5e-05, "s": "x,y} \"q\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}}]}, "tags": ["single-call", "numeric-arguments", "string-escapes", "nested-json", "x-key-with-space", "x-dictsort-order"], "notes": "Same input as the edge-value render in docs/formats/gemma4.md: a key containing spaces (unquoted), a Python-formatted float (1.5e-05), ',', '}' and '\"' inside a string, and case-insensitive key order. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/marker-in-arguments expected_match, stream_equals_nonstream, no_leakage
Checks that did not pass for gemma4/marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchone failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchspecial failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchtoken failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchrand:1:8 failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchrand:2:8 failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchrand:3:8 failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchrand:4:8 failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
expected_matchrand:5:8 failcontent: expected None, got ' and close thoughts with .<|"|>}'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text": "Wrap calls as <|tool_call>call:f{"}'
stream_equals_nonstreamone failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamspecial failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamtoken failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamrand:1:8 failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamrand:2:8 failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamrand:3:8 failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamrand:4:8 failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
stream_equals_nonstreamrand:5:8 failcontent: nonstream None, got ' and close thoughts with .<|"|>}'
no_leakageone failcontent contains '<|"|>'
no_leakagespecial failcontent contains '<|"|>'
no_leakagetoken failcontent contains '<|"|>'
no_leakagerand:1:8 failcontent contains '<|"|>'
no_leakagerand:2:8 failcontent contains '<|"|>'
no_leakagerand:3:8 failcontent contains '<|"|>'
no_leakagerand:4:8 failcontent contains '<|"|>'
no_leakagerand:5:8 failcontent contains '<|"|>'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/edge.jsonl --id gemma4/marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
+        "text": "Wrap calls as <|tool_call>call:f{"
       },
       "name": "echo"

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": " and close thoughts with .<|\"|>}",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
+        "text": "Wrap calls as <|tool_call>call:f{"
       },
       "name": "echo"

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<|tool_call>call:echo{text:<|"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/marker-in-arguments", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<|tool_call>call:echo{text:<|\"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17454, 236782, 1005, 236787, 52, 47352, 9139, 618, 236743, 48, 6639, 236787, 236760, 16454, 49, 532, 3107, 12018, 607, 236743, 101, 236761, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "The string contains the text of the call/channel markers, which the official tokenizer maps to their special-token ids. Inside a <|\"|>-delimited string they are literal text. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/multi-turn-second-call expected_match
Checks that did not pass for gemma4/multi-turn-second-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchone failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchtoken failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"query": "café Zürich", "filters": {"open_now": true}}', got '{"filters": {"open_now": "true"}, "query": "café Zürich"}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/multiturn.jsonl --id gemma4/multi-turn-second-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/gemma4/multiturn.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -6,5 +6,5 @@
       "arguments": {
         "filters": {
-          "open_now": true
+          "open_now": "true"
         },
         "query": "café Zürich"

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, multi-turn, unicode, nested-json.

Raw output

<|tool_call>call:search{filters:{open_now:true},query:<|"|>café Zürich<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "open_now": true
        },
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/multi-turn-second-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|tool_call>call:search{filters:{open_now:true},query:<|\"|>café Zürich<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 2305, 236782, 28881, 29616, 5265, 236779, 2998, 236787, 3397, 1263, 3278, 236787, 52, 123125, 236859, 116315, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich", "filters": {"open_now": true}}}]}, "tags": ["single-call", "multi-turn", "unicode", "nested-json"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/nested-objects-and-arrays expected_match
Checks that did not pass for gemma4/nested-objects-and-arrays
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchone failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchspecial failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchtoken failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": "false"}, {"email": "bo@example.com", "name": "Bo", "optional": "true", "roles": ["no...

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/edge.jsonl --id gemma4/nested-objects-and-arrays --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,10 +9,10 @@
             "email": "ana@example.com",
             "name": "Ana",
-            "optional": false
+            "optional": "false"
           },
           {
             "email": "bo@example.com",
             "name": "Bo",
-            "optional": true,
+            "optional": "true",
             "roles": [
               "notes",
@@ -24,9 +24,9 @@
         "reminders": [
           [
-            10,
+            "10",
             "email"
           ],
           [
-            1,
+            "1",
             "popup"
           ]

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, nested-json.

Raw output

<|tool_call>call:create_event{attendees:[{email:<|"|>ana@example.com<|"|>,name:<|"|>Ana<|"|>,optional:false},{email:<|"|>bo@example.com<|"|>,name:<|"|>Bo<|"|>,optional:true,roles:[<|"|>notes<|"|>,<|"|>timekeeper<|"|>]}],location:null,reminders:[[10,<|"|>email<|"|>],[1,<|"|>popup<|"|>]],title:<|"|>Design review<|"|>,when:{end:<|"|>2026-10-01T10:00:00Z<|"|>,start:<|"|>2026-10-01T09:00:00Z<|"|>,tz:<|"|>UTC<|"|>}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "email": "ana@example.com",
            "name": "Ana",
            "optional": false
          },
          {
            "email": "bo@example.com",
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes",
              "timekeeper"
            ]
          }
        ],
        "location": null,
        "reminders": [
          [
            10,
            "email"
          ],
          [
            1,
            "popup"
          ]
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z",
          "tz": "UTC"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/nested-objects-and-arrays", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}, "reminders": {"type": "array"}, "location": {"anyOf": [{"type": "object"}, {"type": "null"}], "nullable": true}}, "required": ["title", "when"]}}}], "raw_output": "<|tool_call>call:create_event{attendees:[{email:<|\"|>ana@example.com<|\"|>,name:<|\"|>Ana<|\"|>,optional:false},{email:<|\"|>bo@example.com<|\"|>,name:<|\"|>Bo<|\"|>,optional:true,roles:[<|\"|>notes<|\"|>,<|\"|>timekeeper<|\"|>]}],location:null,reminders:[[10,<|\"|>email<|\"|>],[1,<|\"|>popup<|\"|>]],title:<|\"|>Design review<|\"|>,when:{end:<|\"|>2026-10-01T10:00:00Z<|\"|>,start:<|\"|>2026-10-01T09:00:00Z<|\"|>,tz:<|\"|>UTC<|\"|>}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 3663, 236779, 5106, 236782, 102830, 4644, 24845, 236782, 6774, 236787, 52, 1925, 236940, 8358, 236761, 854, 52, 236764, 1201, 236787, 52, 76678, 52, 236764, 31162, 236787, 4530, 49365, 6774, 236787, 52, 1709, 236940, 8358, 236761, 854, 52, 236764, 1201, 236787, 52, 8444, 52, 236764, 31162, 236787, 3397, 236764, 49164, 24845, 52, 23184, 52, 236764, 52, 2289, 46815, 52, 21111, 1604, 7125, 236787, 6455, 236764, 2184, 70593, 24845, 236840, 236770, 236771, 236764, 52, 6774, 52, 41564, 236770, 236764, 52, 22065, 52, 36878, 3250, 236787, 52, 17399, 3540, 52, 236764, 14730, 29616, 643, 236787, 52, 236778, 236771, 236778, 236825, 236772, 236770, 236771, 236772, 236771, 236770, 236774, 236770, 236771, 236787, 236771, 236771, 236787, 236771, 236771, 236953, 52, 236764, 3041, 236787, 52, 236778, 236771, 236778, 236825, 236772, 236770, 236771, 236772, 236771, 236770, 236774, 236771, 236819, 236787, 236771, 236771, 236787, 236771, 236771, 236953, 52, 236764, 21448, 236787, 52, 13902, 52, 1807, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "ana@example.com", "optional": false}, {"name": "Bo", "email": "bo@example.com", "optional": true, "roles": ["notes", "timekeeper"]}], "reminders": [[10, "email"], [1, "popup"]], "location": null}}]}, "tags": ["single-call", "nested-json"], "notes": "Nested keys are unquoted; nested lists of objects and a null value. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/parallel-two-calls-thinking expected_match
Checks that did not pass for gemma4/parallel-two-calls-thinking
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchone failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchspecial failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchtoken failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchrand:1:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchrand:2:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchrand:3:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchrand:4:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'
expected_matchrand:5:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"filters": {"max": "3", "tags": ["a", "b"]}, "query": "café \\"best\\""}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/parallel.jsonl --id gemma4/parallel-two-calls-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/gemma4/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -13,5 +13,5 @@
       "arguments": {
         "filters": {
-          "max": 3,
+          "max": "3",
           "tags": [
             "a",

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<|channel>thought
I should call the tools.
<channel|><|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:<|"|>café "best"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/parallel-two-calls-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|channel>thought\nI should call the tools.\n<channel|><|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:<|\"|>café \"best\"<|\"|>}<tool_call|>", "output_token_ids": [100, 45518, 107, 236777, 1374, 2246, 506, 6436, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787, 52, 123125, 236859, 623, 9783, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "T<|\"|>}}<tool|><turn|>\n<|turn>user\nUse the tools to answer.<turn|>\n<|turn>model\n",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/sglang-nested-array-with-spaces expected_match
Checks that did not pass for gemma4/sglang-nested-array-with-spaces
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchone failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchspecial failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchtoken failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"location": "New York", "nested": [1, 2, {"inner": "val"}]}', got '{"location": "New York", "nested": ["1", "2", {"inner": "val"}]}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-nested-array-with-spaces --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 15 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -7,6 +7,6 @@
         "location": "New York",
         "nested": [
-          1,
-          2,
+          "1",
+          "2",
           {
             "inner": "val"

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5579. Tags: single-call, nested-json.

Raw output

<|tool_call>call:get_weather{location:<|"|>New York<|"|>,nested:[1, 2, {inner:<|"|>val<|"|>}]}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "New York",
        "nested": [
          1,
          2,
          {
            "inner": "val"
          }
        ]
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-nested-array-with-spaces", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5579", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "nested": {"type": "array"}, "details": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:get_weather{location:<|\"|>New York<|\"|>,nested:[1, 2, {inner:<|\"|>val<|\"|>}]}<tool_call|>", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 4199, 3773, 52, 236764, 103895, 24845, 236770, 236764, 236743, 236778, 236764, 642, 3859, 236787, 52, 1111, 52, 165075, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "New York", "nested": [1, 2, {"inner": "val"}]}}]}, "tags": ["single-call", "nested-json"], "notes": "The streamed chunks of the test, joined. Array items are separated by ', ' (not the template's ','). output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/sglang-nested-object expected_match
Checks that did not pass for gemma4/sglang-nested-object
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchone failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchspecial failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchtoken failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}', got '{"location": "Tokyo", "details": {"temp": "25", "unit": "celsius"}}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-nested-object --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 16 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -6,5 +6,5 @@
       "arguments": {
         "details": {
-          "temp": 25,
+          "temp": "25",
           "unit": "celsius"
         },

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5618. Tags: single-call, nested-json, numeric-arguments.

Raw output

<|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>,details:{temp:25,unit:<|"|>celsius<|"|>}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "details": {
          "temp": 25,
          "unit": "celsius"
        },
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-nested-object", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5618", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "nested": {"type": "array"}, "details": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>,details:{temp:25,unit:<|\"|>celsius<|\"|>}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236764, 13308, 29616, 7617, 236787, 236778, 236810, 236764, 6805, 236787, 52, 236755, 42024, 52, 1807, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo", "details": {"temp": 25, "unit": "celsius"}}}]}, "tags": ["single-call", "nested-json", "numeric-arguments"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/sglang-text-around-call expected_match, stream_equals_nonstream
Checks that did not pass for gemma4/sglang-text-around-call
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Some text before after', got 'Some text before'
stream_equals_nonstreamone failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamspecial failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamtoken failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'Some text before', got 'Some text before after'
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'Some text before', got 'Some text before after'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-around-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 14 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before  after",
+  "content": "Some text before",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546. Tags: single-call, text-before-call, text-after-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|> after

Expected parse

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-text-around-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|> after", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49, 1308], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before  after", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "text-after-call"], "notes": "The streamed chunks of the test, joined. Content is all text outside the call. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/text-after-call expected_match, stream_equals_nonstream
Checks that did not pass for gemma4/text-after-call
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Let me check the current conditions in London.', got None
stream_equals_nonstreamone failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamspecial failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamtoken failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:1:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:2:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:3:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:4:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:5:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/basic.jsonl --id gemma4/text-after-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, text-after-call.

Raw output

<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|>Let me check the current conditions in London.

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/text-after-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|>Let me check the current conditions in London.", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 6481, 786, 2426, 506, 1873, 3439, 528, 5860, 236761], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-after-call"], "notes": "The official template renders an assistant message's text content AFTER its tool calls. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/text-after-call-thinking expected_match, stream_equals_nonstream
Checks that did not pass for gemma4/text-after-call-thinking
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Let me check the current conditions in London.', got None
stream_equals_nonstreamone failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamspecial failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamtoken failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:1:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:2:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:3:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:4:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'
stream_equals_nonstreamrand:5:8 failcontent: nonstream None, got 'Let me check the current conditions in London.'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/basic.jsonl --id gemma4/text-after-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": "I need live weather data for London.",
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, text-after-call, reasoning.

Raw output

<|channel>thought
I need live weather data for London.
<channel|><|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|>Let me check the current conditions in London.

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/text-after-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nI need live weather data for London.\n<channel|><|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|>Let me check the current conditions in London.", "output_token_ids": [100, 45518, 107, 236777, 1202, 3892, 7606, 1262, 573, 5860, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 6481, 786, 2426, 506, 1873, 3439, 528, 5860, 236761], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-after-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "T<|\"|>}}<tool|><turn|>\n<|turn>user\nUse the tools to answer.<turn|>\n<|turn>model\n",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/truncated-before-close expected_error
Checks that did not pass for gemma4/truncated-before-close
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-before-close --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris<|"|>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-before-close", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429, 52], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Cut before the closing '}' and <tool_call|>. Token prefix of the full render, cut right after 'Paris<|\"|>' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/truncated-in-name expected_error, arguments_schema
Checks that did not pass for gemma4/truncated-in-name
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get'] for Output cut by max_tokens inside the tool call: no complete call exists.
arguments_schemanonstream fail[0] 'get' is not an offered tool
arguments_schemaone fail[0] 'get' is not an offered tool
arguments_schemaspecial fail[0] 'get' is not an offered tool
arguments_schematoken fail[0] 'get' is not an offered tool
arguments_schemarand:1:8 fail[0] 'get' is not an offered tool
arguments_schemarand:2:8 fail[0] 'get' is not an offered tool
arguments_schemarand:3:8 fail[0] 'get' is not an offered tool
arguments_schemarand:4:8 fail[0] 'get' is not an offered tool
arguments_schemarand:5:8 fail[0] 'get' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-in-name --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-in-name", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get", "output_token_ids": [48, 6639, 236787, 828], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after 'call:get' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/truncated-mid-string expected_error
Checks that did not pass for gemma4/truncated-mid-string
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-mid-string --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-mid-string", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after 'Par' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/truncated-second-parallel-call expected_match
Checks that did not pass for gemma4/truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/gemma4/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,17 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "filters": {
+          "max": "3",
+          "tags": [
+            "a",
+            "b"
+          ]
+        },
+        "query": ""
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated, parallel-calls, unicode.

Raw output

<|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-second-parallel-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut. A correct parser keeps the complete call only. Token prefix of the full render, cut right after 'query:' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
fail gemma4/vllm-malformed-no-brace expected_error, arguments_schema
Checks that did not pass for gemma4/vllm-malformed-no-brace
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorone failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorspecial failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errortoken failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorrand:1:8 failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorrand:2:8 failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorrand:3:8 failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorrand:4:8 failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
expected_errorrand:5:8 failreturned 1 tool call(s) ['bad_func no brace'] for The call has no '{...}' argument object and names no offered tool; it is not a valid call.
arguments_schemanonstream fail[0] 'bad_func no brace' is not an offered tool
arguments_schemaone fail[0] 'bad_func no brace' is not an offered tool
arguments_schemaspecial fail[0] 'bad_func no brace' is not an offered tool
arguments_schematoken fail[0] 'bad_func no brace' is not an offered tool
arguments_schemarand:1:8 fail[0] 'bad_func no brace' is not an offered tool
arguments_schemarand:2:8 fail[0] 'bad_func no brace' is not an offered tool
arguments_schemarand:3:8 fail[0] 'bad_func no brace' is not an offered tool
arguments_schemarand:4:8 fail[0] 'bad_func no brace' is not an offered tool
arguments_schemarand:5:8 fail[0] 'bad_func no brace' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/malformed.jsonl --id gemma4/vllm-malformed-no-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/gemma4/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "bad_func no brace"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942. Tags: malformed.

Expected graceful failure: The call has no '{...}' argument object and names no offered tool; it is not a valid call. (accept: no_tool_calls, content_passthrough).

Raw output

<|tool_call>call:bad_func no brace<tool_call|>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/vllm-malformed-no-brace", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "<|tool_call>call:bad_func no brace<tool_call|>", "output_token_ids": [48, 6639, 236787, 15242, 236779, 6823, 951, 24088, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "The call has no '{...}' argument object and names no offered tool; it is not a valid call.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed"], "notes": "vLLM's test only asserts that the recovered name stays bounded by <tool_call|> (it returns a call named 'bad_func no brace'); per the format there is no valid call here. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
soft pass gemma4/sglang-text-before-call expected_match, stream_equals_nonstream
Checks that did not pass for gemma4/sglang-text-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Some text before ', got 'Some text before'
stream_equals_nonstreamone soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamspecial soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamrand:1:8 soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamrand:3:8 soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '
stream_equals_nonstreamrand:5:8 soft passcontent (whitespace only): nonstream 'Some text before', got 'Some text before '

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 13 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before ",
+  "content": "Some text before",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Some text before ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5535. Tags: single-call, text-before-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|>

Expected parse

{
  "content": "Some text before ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5535", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|>", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}
soft pass gemma4/vllm-text-before-call expected_match, stream_equals_nonstream
Checks that did not pass for gemma4/vllm-text-before-call
CheckStrategyResultDetail
expected_matchone soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchspecial soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamone soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamspecial soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamrand:1:8 soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamrand:3:8 soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '
stream_equals_nonstreamrand:5:8 soft passcontent (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323. Tags: single-call, text-before-call.

Raw output

Let me check the weather for you. <|tool_call>call:get_weather{location:<|"|>Paris<|"|>}<tool_call|>

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/vllm-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Let me check the weather for you. <|tool_call>call:get_weather{location:<|\"|>Paris<|\"|>}<tool_call|>", "output_token_ids": [6481, 786, 2426, 506, 7606, 573, 611, 236761, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 50429, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "text-before-call"], "notes": "The template renders text after calls, but models also emit text before them. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "google/gemma-4-31B-it",
  "model_rule": "gemma-4",
  "model_type": "gemma4",
  "parser_source": "docs/formats/gemma4.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "n>user\nUse the tools to answer.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
  "reasoning_parser": "gemma4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "google/gemma-4-31B-it",
    "requested_mode": "auto",
    "resolved_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "tokenizer_class": "transformers.models.gemma.tokenization_gemma.GemmaTokenizer"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "gemma4",
  "units_source": "fixture.output_token_ids"
}