Gemma 4 (<|tool_call>call:NAME{...} object notation) on sglang 0.5.20

fail 62% strict pass 30 pass · 5 soft pass · 13 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	30	5	9	68%
`expected_error`	2	0	2	50%
`stream_equals_nonstream`	38	0	10	79%
`split_invariance`	48	0	0	100%
`no_leakage`	48	0	0	100%
`arguments_json`	35	0	4	90%
`arguments_schema`	33	0	6	85%
`parallel_order`	3	0	1	75%

Fixtures needing attention

fail gemma4/edge-values-key-with-space expected_match

Checks that did not pass for gemma4/edge-values-key-with-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"key with space": 1.5e-05, "s": "x,y} \\"q\\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}', got '{"flag": false, "key with space": 1.5e-05, "none": "null", "obj": {"a": [1, "t"], "B": 1}, "s": "x,y} \\"q\\""}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/edge.jsonl --id gemma4/edge-values-key-with-space --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -7,5 +7,5 @@
         "flag": false,
         "key with space": 1.5e-05,
-        "none": null,
+        "none": "null",
         "obj": {
           "B": 1,

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, numeric-arguments, string-escapes, nested-json, x-key-with-space, x-dictsort-order.

Raw output

<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|"|>t<|"|>],B:1},s:<|"|>x,y} "q"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "flag": false,
        "key with space": 1.5e-05,
        "none": null,
        "obj": {
          "B": 1,
          "a": [
            1,
            "t"
          ]
        },
        "s": "x,y} \"q\""
      },
      "name": "record"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/edge-values-key-with-space", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "record", "description": "Store a record with arbitrary keys.", "parameters": {"type": "object", "properties": {"key with space": {"type": "number"}, "s": {"type": "string"}, "flag": {"type": "boolean"}, "none": {"anyOf": [{"type": "string"}, {"type": "null"}], "nullable": true}, "obj": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|\"|>t<|\"|>],B:1},s:<|\"|>x,y} \"q\"<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17789, 236782, 16213, 236787, 4530, 236764, 2478, 607, 2557, 236787, 236770, 236761, 236810, 236744, 236772, 236771, 236810, 236764, 9745, 236787, 6455, 236764, 7837, 29616, 236746, 24845, 236770, 236764, 52, 236745, 52, 1604, 236799, 236787, 236770, 1263, 236751, 236787, 52, 236781, 236764, 236762, 236783, 623, 236809, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "record", "arguments": {"key with space": 1.5e-05, "s": "x,y} \"q\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}}]}, "tags": ["single-call", "numeric-arguments", "string-escapes", "nested-json", "x-key-with-space", "x-dictsort-order"], "notes": "Same input as the edge-value render in docs/formats/gemma4.md: a key containing spaces (unquoted), a Python-formatted float (1.5e-05), ',', '}' and '\"' inside a string, and case-insensitive key order. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/marker-in-arguments expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}', got '{"text": "Wrap calls as <\|tool_call>call:f{}"}'
`stream_equals_nonstream`	`one`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`special`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`token`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls[0].arguments: nonstream '{"text": "Wrap calls as <\|tool_call>call:f{}"}', got '{"text": "Wrap calls as <\|tool_call>call:f{}<tool_call\|> and close thoughts with <channel\|>."}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/edge.jsonl --id gemma4/marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
+        "text": "Wrap calls as <|tool_call>call:f{}"
       },
       "name": "echo"

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<|tool_call>call:echo{text:<|"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/marker-in-arguments", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<|tool_call>call:echo{text:<|\"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17454, 236782, 1005, 236787, 52, 47352, 9139, 618, 236743, 48, 6639, 236787, 236760, 16454, 49, 532, 3107, 12018, 607, 236743, 101, 236761, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "The string contains the text of the call/channel markers, which the official tokenizer maps to their special-token ids. Inside a <|\"|>-delimited string they are literal text. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/nested-objects-and-arrays expected_match, arguments_schema

Checks that did not pass for gemma4/nested-objects-and-arrays
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "an..., got '{"attendees": [{"email": "ana@example.com", "name": "Ana", "optional": false}, {"email": "bo@example.com", "name": "Bo", "optional": true, "roles": ["notes"...
`arguments_schema`	`nonstream`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`one`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`special`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`token`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`rand:1:8`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`rand:2:8`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`rand:3:8`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`rand:4:8`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas
`arguments_schema`	`rand:5:8`	fail	[0] create_event: $.location: 'null' is not valid under any of the given schemas

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/edge.jsonl --id gemma4/nested-objects-and-arrays --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -21,5 +21,5 @@
           }
         ],
-        "location": null,
+        "location": "null",
         "reminders": [
           [

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, nested-json.

Raw output

<|tool_call>call:create_event{attendees:[{email:<|"|>ana@example.com<|"|>,name:<|"|>Ana<|"|>,optional:false},{email:<|"|>bo@example.com<|"|>,name:<|"|>Bo<|"|>,optional:true,roles:[<|"|>notes<|"|>,<|"|>timekeeper<|"|>]}],location:null,reminders:[[10,<|"|>email<|"|>],[1,<|"|>popup<|"|>]],title:<|"|>Design review<|"|>,when:{end:<|"|>2026-10-01T10:00:00Z<|"|>,start:<|"|>2026-10-01T09:00:00Z<|"|>,tz:<|"|>UTC<|"|>}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "email": "ana@example.com",
            "name": "Ana",
            "optional": false
          },
          {
            "email": "bo@example.com",
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes",
              "timekeeper"
            ]
          }
        ],
        "location": null,
        "reminders": [
          [
            10,
            "email"
          ],
          [
            1,
            "popup"
          ]
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z",
          "tz": "UTC"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/nested-objects-and-arrays", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}, "reminders": {"type": "array"}, "location": {"anyOf": [{"type": "object"}, {"type": "null"}], "nullable": true}}, "required": ["title", "when"]}}}], "raw_output": "<|tool_call>call:create_event{attendees:[{email:<|\"|>ana@example.com<|\"|>,name:<|\"|>Ana<|\"|>,optional:false},{email:<|\"|>bo@example.com<|\"|>,name:<|\"|>Bo<|\"|>,optional:true,roles:[<|\"|>notes<|\"|>,<|\"|>timekeeper<|\"|>]}],location:null,reminders:[[10,<|\"|>email<|\"|>],[1,<|\"|>popup<|\"|>]],title:<|\"|>Design review<|\"|>,when:{end:<|\"|>2026-10-01T10:00:00Z<|\"|>,start:<|\"|>2026-10-01T09:00:00Z<|\"|>,tz:<|\"|>UTC<|\"|>}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 3663, 236779, 5106, 236782, 102830, 4644, 24845, 236782, 6774, 236787, 52, 1925, 236940, 8358, 236761, 854, 52, 236764, 1201, 236787, 52, 76678, 52, 236764, 31162, 236787, 4530, 49365, 6774, 236787, 52, 1709, 236940, 8358, 236761, 854, 52, 236764, 1201, 236787, 52, 8444, 52, 236764, 31162, 236787, 3397, 236764, 49164, 24845, 52, 23184, 52, 236764, 52, 2289, 46815, 52, 21111, 1604, 7125, 236787, 6455, 236764, 2184, 70593, 24845, 236840, 236770, 236771, 236764, 52, 6774, 52, 41564, 236770, 236764, 52, 22065, 52, 36878, 3250, 236787, 52, 17399, 3540, 52, 236764, 14730, 29616, 643, 236787, 52, 236778, 236771, 236778, 236825, 236772, 236770, 236771, 236772, 236771, 236770, 236774, 236770, 236771, 236787, 236771, 236771, 236787, 236771, 236771, 236953, 52, 236764, 3041, 236787, 52, 236778, 236771, 236778, 236825, 236772, 236770, 236771, 236772, 236771, 236770, 236774, 236771, 236819, 236787, 236771, 236771, 236787, 236771, 236771, 236953, 52, 236764, 21448, 236787, 52, 13902, 52, 1807, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z", "tz": "UTC"}, "attendees": [{"name": "Ana", "email": "ana@example.com", "optional": false}, {"name": "Bo", "email": "bo@example.com", "optional": true, "roles": ["notes", "timekeeper"]}], "reminders": [[10, "email"], [1, "popup"]], "location": null}}]}, "tags": ["single-call", "nested-json"], "notes": "Nested keys are unquoted; nested lists of objects and a null value. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/numeric-arguments expected_match, arguments_schema

Checks that did not pass for gemma4/numeric-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`one`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`special`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`rand:1:8`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`rand:2:8`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`rand:3:8`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`rand:4:8`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`expected_match`	`rand:5:8`	fail	tool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}', got '{"exact": true, "limit": "null", "tolerance": "1e-05", "values": [0, -0.5, "1e+21"], "x": -3, "y": 2.5}'
`arguments_schema`	`nonstream`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`one`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`special`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`token`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`rand:1:8`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`rand:2:8`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`rand:3:8`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`rand:4:8`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'
`arguments_schema`	`rand:5:8`	fail	[0] calculate: $.limit: 'null' is not valid under any of the given schemas; $.tolerance: '1e-05' is not of type 'number'; $.values[2]: '1e+21' is not of type 'number'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/edge.jsonl --id gemma4/numeric-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -6,10 +6,10 @@
       "arguments": {
         "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
+        "limit": "null",
+        "tolerance": "1e-05",
         "values": [
           0,
           -0.5,
-          1e+21
+          "1e+21"
         ],
         "x": -3,

Fixture

Raw output

<|tool_call>call:calculate{exact:true,limit:null,tolerance:1e-05,values:[0,-0.5,1e+21],x:-3,y:2.5}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/numeric-arguments", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"anyOf": [{"type": "integer"}, {"type": "null"}], "nullable": true}, "values": {"type": "array", "items": {"type": "number"}}}, "required": ["x", "y"]}}}], "raw_output": "<|tool_call>call:calculate{exact:true,limit:null,tolerance:1e-05,values:[0,-0.5,1e+21],x:-3,y:2.5}<tool_call|>", "output_token_ids": [48, 6639, 236787, 30718, 236782, 67655, 236787, 3397, 236764, 17846, 236787, 6455, 236764, 113083, 236787, 236770, 236744, 236772, 236771, 236810, 236764, 7558, 24845, 236771, 10442, 236771, 236761, 236810, 236764, 236770, 236744, 236862, 236778, 236770, 1604, 236781, 16728, 236800, 236764, 236762, 236787, 236778, 236761, 236810, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21]}}]}, "tags": ["single-call", "numeric-arguments", "x-python-float"], "notes": "Numbers are rendered with Python str(): 1e-05 and 1e+21 are not how json.dumps writes them everywhere. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/parallel-three-calls expected_match, stream_equals_nonstream, arguments_json, arguments_schema, parallel_order

Checks that did not pass for gemma4/parallel-three-calls
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream ['get_weather', 'get_weather', 'get_weather'], got ['get_weatherget_weatherget_weather']
`arguments_json`	`one`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`special`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`token`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`rand:1:8`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`rand:2:8`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`rand:3:8`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`rand:4:8`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_json`	`rand:5:8`	fail	[0] get_weatherget_weatherget_weather: arguments are not valid JSON text ('{"city": "Paris"}{"city": "London"}{"city": "Tokyo", "unit": "c"}': Extra data: line 1 column 18 (char 17))
`arguments_schema`	`one`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`special`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`token`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`rand:1:8`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`rand:2:8`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`rand:3:8`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`rand:4:8`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`arguments_schema`	`rand:5:8`	fail	[0] 'get_weatherget_weatherget_weather' is not an offered tool
`parallel_order`	`one`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`special`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`token`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`rand:1:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`rand:2:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`rand:3:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`rand:4:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']
`parallel_order`	`rand:5:8`	fail	expected 3 calls ['get_weather', 'get_weather', 'get_weather'], got 1 ['get_weatherget_weatherget_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/parallel.jsonl --id gemma4/parallel-three-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/gemma4/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,20 +5,7 @@
     {
       "arguments": {
-        "city": "Paris"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\"}{\"city\": \"London\"}{\"city\": \"Tokyo\", \"unit\": \"c\"}"
       },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "c"
-      },
-      "name": "get_weather"
+      "name": "get_weatherget_weatherget_weather"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: parallel-calls.

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris<|"|>}<tool_call|><|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|><|tool_call>call:get_weather{city:<|"|>Tokyo<|"|>,unit:<|"|>c<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/parallel-three-calls", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>}<tool_call|><|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|><|tool_call>call:get_weather{city:<|\"|>Tokyo<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429, 52, 236783, 49, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 89265, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}, {"name": "get_weather", "arguments": {"city": "London"}}, {"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "c"}}]}, "tags": ["parallel-calls"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/sglang-text-around-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/sglang-text-around-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Some text before after', got 'Some text before '
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Some text before ', got 'Some text before after'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Some text before ', got 'Some text before after'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-around-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before  after",
+  "content": "Some text before ",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546. Tags: single-call, text-before-call, text-after-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|> after

Expected parse

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/sglang-text-around-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|> after", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49, 1308], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before  after", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "text-after-call"], "notes": "The streamed chunks of the test, joined. Content is all text outside the call. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/text-after-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/text-after-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Let me check the current conditions in London.', got None
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/basic.jsonl --id gemma4/text-after-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, text-after-call.

Raw output

<|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|>Let me check the current conditions in London.

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/text-after-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|>Let me check the current conditions in London.", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 6481, 786, 2426, 506, 1873, 3439, 528, 5860, 236761], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-after-call"], "notes": "The official template renders an assistant message's text content AFTER its tool calls. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/text-after-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/text-after-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Let me check the current conditions in London.', got None; reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`stream_equals_nonstream`	`one`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`special`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream None, got 'Let me check the current conditions in London.'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/basic.jsonl --id gemma4/text-after-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "content": null,
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, text-after-call, reasoning.

Raw output

<|channel>thought
I need live weather data for London.
<channel|><|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|>Let me check the current conditions in London.

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/text-after-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nI need live weather data for London.\n<channel|><|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|>Let me check the current conditions in London.", "output_token_ids": [100, 45518, 107, 236777, 1202, 3892, 7606, 1262, 573, 5860, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 6481, 786, 2426, 506, 1873, 3439, 528, 5860, 236761], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-after-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      105,
      4368,
      107
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/truncated-before-close expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for gemma4/truncated-before-close
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris<\|"\|>', got None; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-before-close --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris<|"|>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/truncated-before-close", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429, 52], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Cut before the closing '}' and <tool_call|>. Token prefix of the full render, cut right after 'Paris<|\"|>' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/truncated-in-name stream_equals_nonstream

Checks that did not pass for gemma4/truncated-in-name
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|tool_call>call:get', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|tool_call>call:get', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-in-name --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<|tool_call>call:get",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/truncated-in-name", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get", "output_token_ids": [48, 6639, 236787, 828], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after 'call:get' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/truncated-mid-string expected_error, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for gemma4/truncated-mid-string
Check	Strategy	Result	Detail
`expected_error`	`one`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`special`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:1:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:2:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:3:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:4:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`expected_error`	`rand:5:8`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|tool_call>call:get_weather{city:<\|"\|>Paris', got None; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`one`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:1:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:2:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:3:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:5:8`	fail	[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-mid-string --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<|tool_call>call:get_weather{city:<|\"|>Paris",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/truncated-mid-string", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after 'Par' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gemma4/truncated-second-parallel-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for gemma4/truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`one`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`special`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:1:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:2:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:3:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:4:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`stream_equals_nonstream`	`rand:5:8`	fail	tool_calls: nonstream ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`one`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`special`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`token`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:1:8`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:2:8`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:3:8`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:4:8`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_json`	`rand:5:8`	fail	[1] search: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
`arguments_schema`	`one`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`special`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:1:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:2:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:3:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:4:8`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`rand:5:8`	fail	[1] search: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/gemma4/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
+      "name": "search"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated, parallel-calls, unicode.

Raw output

<|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/truncated-second-parallel-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut. A correct parser keeps the complete call only. Token prefix of the full render, cut right after 'query:' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail gemma4/vllm-malformed-no-brace stream_equals_nonstream

Checks that did not pass for gemma4/vllm-malformed-no-brace
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|tool_call>call:bad_func no brace<tool_call\|>', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/malformed.jsonl --id gemma4/vllm-malformed-no-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/gemma4/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "<|tool_call>call:bad_func no brace<tool_call|>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942. Tags: malformed.

Expected graceful failure: The call has no '{...}' argument object and names no offered tool; it is not a valid call. (accept: no_tool_calls, content_passthrough).

Raw output

<|tool_call>call:bad_func no brace<tool_call|>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-malformed-no-brace", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "<|tool_call>call:bad_func no brace<tool_call|>", "output_token_ids": [48, 6639, 236787, 15242, 236779, 6823, 951, 24088, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "The call has no '{...}' argument object and names no offered tool; it is not a valid call.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed"], "notes": "vLLM's test only asserts that the recovered name stays bounded by <tool_call|> (it returns a call named 'bad_func no brace'); per the format there is no valid call here. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass gemma4/empty-arguments-thinking expected_match

Checks that did not pass for gemma4/empty-arguments-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/basic.jsonl --id gemma4/empty-arguments-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "get_time takes no arguments.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, empty-arguments, reasoning.

Raw output

<|channel>thought
get_time takes no arguments.
<channel|><|tool_call>call:get_time{}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/empty-arguments-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<|channel>thought\nget_time takes no arguments.\n<channel|><|tool_call>call:get_time{}<tool_call|>", "output_token_ids": [100, 45518, 107, 828, 236779, 2289, 4716, 951, 12130, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 2289, 16454, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "get_time takes no arguments.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      105,
      4368,
      107
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass gemma4/no-call-thinking expected_match

Checks that did not pass for gemma4/no-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/basic.jsonl --id gemma4/no-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Hello! How can I help you today?",
-  "reasoning_content": "The user is just greeting me. No tool is needed.",
+  "reasoning_content": "The user is just greeting me. No tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: no-call, reasoning.

Raw output

<|channel>thought
The user is just greeting me. No tool is needed.
<channel|>Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/no-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nThe user is just greeting me. No tool is needed.\n<channel|>Hello! How can I help you today?", "output_token_ids": [100, 45518, 107, 818, 2430, 563, 1164, 26227, 786, 236761, 2301, 5904, 563, 4354, 236761, 107, 101, 9259, 236888, 2088, 740, 564, 1601, 611, 3124, 236881], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "The user is just greeting me. No tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      105,
      4368,
      107
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass gemma4/parallel-two-calls-thinking expected_match

Checks that did not pass for gemma4/parallel-two-calls-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/parallel.jsonl --id gemma4/parallel-two-calls-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/gemma4/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<|channel>thought
I should call the tools.
<channel|><|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:<|"|>café "best"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/parallel-two-calls-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|channel>thought\nI should call the tools.\n<channel|><|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:<|\"|>café \"best\"<|\"|>}<tool_call|>", "output_token_ids": [100, 45518, 107, 236777, 1374, 2246, 506, 6436, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787, 52, 123125, 236859, 623, 9783, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      105,
      4368,
      107
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

soft pass gemma4/single-call-thinking expected_match

Checks that did not pass for gemma4/single-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`one`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`special`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:1:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:2:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:3:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:4:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`expected_match`	`rand:5:8`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/basic.jsonl --id gemma4/single-call-thinking --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, reasoning.

Raw output

<|channel>thought
The user wants the current weather in Paris. I'll call get_weather.
<channel|><|tool_call>call:get_weather{city:<|"|>Paris<|"|>,unit:<|"|>c<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/single-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nThe user wants the current weather in Paris. I'll call get_weather.\n<channel|><|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|>", "output_token_ids": [100, 45518, 107, 818, 2430, 8150, 506, 1873, 7606, 528, 9079, 236761, 564, 236789, 859, 2246, 974, 236779, 19323, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      105,
      4368,
      107
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": true,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

soft pass gemma4/vllm-text-before-call expected_match

Checks that did not pass for gemma4/vllm-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`one`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`special`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`rand:1:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`rand:2:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`rand:3:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`rand:4:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`expected_match`	`rand:5:8`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323. Tags: single-call, text-before-call.

Raw output

Let me check the weather for you. <|tool_call>call:get_weather{location:<|"|>Paris<|"|>}<tool_call|>

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Let me check the weather for you. <|tool_call>call:get_weather{location:<|\"|>Paris<|\"|>}<tool_call|>", "output_token_ids": [6481, 786, 2426, 506, 7606, 573, 611, 236761, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 50429, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "text-before-call"], "notes": "The template renders text after calls, but models also emit text before them. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gemma4",
    "tool_call_parser": "gemma4"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "Gemma4ForConditionalGeneration"
    ],
    "model_type": "gemma4"
  },
  "hf_config_error": null,
  "model": "google/gemma-4-31B-it",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      107,
      100,
      45518,
      107,
      101
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "Gemma4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "gemma4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<eos>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "GemmaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "google/gemma-4-31B-it",
    "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Gemma4Detector",
  "tool_call_parser": "gemma4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model