Gemma 4 (<|tool_call>call:NAME{...} object notation) on transformers 5.17.0

fail 69% strict pass 33 pass · 7 soft pass · 8 fail

Run

Engine

transformers 5.17.0

Run

2026-09-26T05:18:44Z to 2026-09-26T05:18:47Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "huggingface_hub": "1.33.0",
  "jinja2": "3.1.6",
  "jmespath": "1.1.0",
  "pinned_repos": {
    "google/gemma-4-12B-it": "707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7",
    "google/gemma-4-26B-A4B-it": "4d7ae4984b7db7de8f8457170b3f1a419ee76d52",
    "google/gemma-4-31B-it": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "google/gemma-4-E2B-it": "3e22461f65e89153144f8adb70e3b8c2cc9845a7",
    "google/gemma-4-E4B-it": "ee0ef6023621cff504d758262d4e04895a5af4a2"
  },
  "pinned_version": "5.17.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "one",
  "torch": null
}

Results file

data/transformers-5.17.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Soft	Fail	Strict pass rate
`expected_match`	30	7	7	68%
`expected_error`	3	0	1	75%
`stream_equals_nonstream`	39	7	2	81%
`no_leakage`	48	0	0	100%
`arguments_json`	32	0	0	100%
`arguments_schema`	32	0	0	100%
`parallel_order`	4	0	0	100%

8 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail gemma4/bug-ollama-18390-key-with-spaces expected_match

Checks that did not pass for gemma4/bug-ollama-18390-key-with-spaces
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{name:<\|"\|>Demo<\|"\|>, connections:{Basic LLM Chain:{main:[[{node:<\|"\|>X<\|"\|>, type:...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{name:<\|"\|>Demo<\|"\|>, connections:{Basic LLM Chain:{main:[[{node:<\|"\|>X<\|"\|>, type:...

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/bug-reports.jsonl --id gemma4/bug-ollama-18390-key-with-spaces --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 1 of fixtures/gemma4/bug-reports.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,25 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '{name:<|\"|>Demo<|\"|>, connections:{Basic LLM Chain:{main:[[{node:<|\"|>X<|\"|>, type:<|\"|>main<|\"|>, index:0}]]}}}'\nTransformed: '{\"name\":\"Demo\", connections:{Basic LLM Chain:{\"main\":[[{\"node\":\"X\", type:\"main\", index:0}]]}}}'\nError: Expecting property name enclosed in double quotes: line 1 column 17 (char 16)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "connections": {
-          "Basic LLM Chain": {
-            "main": [
-              [
-                {
-                  "index": 0,
-                  "node": "X",
-                  "type": "main"
-                }
-              ]
-            ]
-          }
-        },
-        "name": "Demo"
-      },
-      "name": "create_workflow"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18390. Tags: single-call, nested-json, regression, x-key-with-space.

Raw output

<|tool_call>call:create_workflow{name:<|"|>Demo<|"|>, connections:{Basic LLM Chain:{main:[[{node:<|"|>X<|"|>, type:<|"|>main<|"|>, index:0}]]}}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "connections": {
          "Basic LLM Chain": {
            "main": [
              [
                {
                  "index": 0,
                  "node": "X",
                  "type": "main"
                }
              ]
            ]
          }
        },
        "name": "Demo"
      },
      "name": "create_workflow"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/bug-ollama-18390-key-with-spaces", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18390", "revision": "issue body (2026-09-11)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "create_workflow", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "connections": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:create_workflow{name:<|\"|>Demo<|\"|>, connections:{Basic LLM Chain:{main:[[{node:<|\"|>X<|\"|>, type:<|\"|>main<|\"|>, index:0}]]}}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 3663, 236779, 95778, 236782, 1201, 236787, 52, 33846, 52, 236764, 13792, 29616, 18397, 32050, 236792, 35211, 29616, 3241, 24845, 53103, 5335, 236787, 52, 236917, 52, 236764, 1722, 236787, 52, 3241, 52, 236764, 3546, 236787, 236771, 236783, 10660, 19465, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_workflow", "arguments": {"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}}]}, "tags": ["single-call", "nested-json", "regression", "x-key-with-space"], "notes": "The 'bare key with spaces' input of the issue's parser-level Go test. Ollama dropped the whole call. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/edge-values-key-with-space expected_match

Checks that did not pass for gemma4/edge-values-key-with-space
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<\|"\|>t<\|"\|>],B:1},s:<\|"\|>x,y...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<\|"\|>t<\|"\|>],B:1},s:<\|"\|>x,y...

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/edge.jsonl --id gemma4/edge-values-key-with-space --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 2 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,22 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|\"|>t<|\"|>],B:1},s:<|\"|>x,y} \"q\"<|\"|>}'\nTransformed: '{\"flag\":false,key with space:1.5e-05,\"none\":null,\"obj\":{\"a\":[1,\"t\"],\"B\":1},\"s\":\"x,y} \\\\\"q\\\\\"\"}'\nError: Expecting property name enclosed in double quotes: line 1 column 15 (char 14)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "flag": false,
-        "key with space": 1.5e-05,
-        "none": null,
-        "obj": {
-          "B": 1,
-          "a": [
-            1,
-            "t"
-          ]
-        },
-        "s": "x,y} \"q\""
-      },
-      "name": "record"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, numeric-arguments, string-escapes, nested-json, x-key-with-space, x-dictsort-order.

Raw output

<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|"|>t<|"|>],B:1},s:<|"|>x,y} "q"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "flag": false,
        "key with space": 1.5e-05,
        "none": null,
        "obj": {
          "B": 1,
          "a": [
            1,
            "t"
          ]
        },
        "s": "x,y} \"q\""
      },
      "name": "record"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/edge-values-key-with-space", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "record", "description": "Store a record with arbitrary keys.", "parameters": {"type": "object", "properties": {"key with space": {"type": "number"}, "s": {"type": "string"}, "flag": {"type": "boolean"}, "none": {"anyOf": [{"type": "string"}, {"type": "null"}], "nullable": true}, "obj": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|\"|>t<|\"|>],B:1},s:<|\"|>x,y} \"q\"<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17789, 236782, 16213, 236787, 4530, 236764, 2478, 607, 2557, 236787, 236770, 236761, 236810, 236744, 236772, 236771, 236810, 236764, 9745, 236787, 6455, 236764, 7837, 29616, 236746, 24845, 236770, 236764, 52, 236745, 52, 1604, 236799, 236787, 236770, 1263, 236751, 236787, 52, 236781, 236764, 236762, 236783, 623, 236809, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "record", "arguments": {"key with space": 1.5e-05, "s": "x,y} \"q\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}}]}, "tags": ["single-call", "numeric-arguments", "string-escapes", "nested-json", "x-key-with-space", "x-dictsort-order"], "notes": "Same input as the edge-value render in docs/formats/gemma4.md: a key containing spaces (unquoted), a Python-formatted float (1.5e-05), ',', '}' and '\"' inside a string, and case-insensitive key order. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/marker-in-arguments expected_match

Checks that did not pass for gemma4/marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{text:<\|"\|>Wrap calls as <\|tool_call>call:f{}\'\nTransformed: \'{"text":<\|"\|>Wrap c...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{text:<\|"\|>Wrap calls as <\|tool_call>call:f{}\'\nTransformed: \'{"text":<\|"\|>Wrap c...

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/edge.jsonl --id gemma4/marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 4 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '{text:<|\"|>Wrap calls as <|tool_call>call:f{}'\nTransformed: '{\"text\":<|\"|>Wrap calls as <|tool_call>call:f{}'\nError: Expecting value: line 1 column 9 (char 8)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<|tool_call>call:echo{text:<|"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/marker-in-arguments", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<|tool_call>call:echo{text:<|\"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17454, 236782, 1005, 236787, 52, 47352, 9139, 618, 236743, 48, 6639, 236787, 236760, 16454, 49, 532, 3107, 12018, 607, 236743, 101, 236761, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "The string contains the text of the call/channel markers, which the official tokenizer maps to their special-token ids. Inside a <|\"|>-delimited string they are literal text. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/sglang-text-around-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/sglang-text-around-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Some text before after', got 'after'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'after', got 'Some text before after'

Synthetic strategies (not counted): stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-around-call --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 14 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before  after",
+  "content": "after",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546. Tags: single-call, text-before-call, text-after-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|> after

Expected parse

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/sglang-text-around-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|> after", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49, 1308], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before  after", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "text-after-call"], "notes": "The streamed chunks of the test, joined. Content is all text outside the call. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/truncated-second-parallel-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{filters:{max:3,tags:[<\|"\|>a<\|"\|>,<\|"\|>b<\|"\|>]},query:\'\nTransformed: \'{"filters"...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'{filters:{max:3,tags:[<\|"\|>a<\|"\|>,<\|"\|>b<\|"\|>]},query:\'\nTransformed: \'{"filters"...
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-second-parallel-call --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 5 of fixtures/gemma4/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:'\nTransformed: '{\"filters\":{\"max\":3,\"tags\":[\"a\",\"b\"]},\"query\":'\nError: Expecting value: line 1 column 47 (char 46)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:'\nTransformed: '{\"filters\":{\"max\":3,\"tags\":[\"a\",\"b\"]},\"query\":'\nError: Expecting value: line 1 column 47 (char 46)",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated, parallel-calls, unicode.

Raw output

<|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/truncated-second-parallel-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut. A correct parser keeps the complete call only. Token prefix of the full render, cut right after 'query:' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/vllm-dotted-function-name expected_match

Checks that did not pass for gemma4/vllm-dotted-function-name
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'.get{location:<\|"\|>London<\|"\|>}\'\nTransformed: \'.get{"location":"London"}\'\nErro...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'.get{location:<\|"\|>London<\|"\|>}\'\nTransformed: \'.get{"location":"London"}\'\nErro...

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-dotted-function-name --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 9 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '.get{location:<|\"|>London<|\"|>}'\nTransformed: '.get{\"location\":\"London\"}'\nError: Expecting value: line 1 column 1 (char 0)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "London"
-      },
-      "name": "weather.get"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L402. Tags: single-call, x-name-punctuation.

Raw output

<|tool_call>call:weather.get{location:<|"|>London<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "London"
      },
      "name": "weather.get"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-dotted-function-name", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L402", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "weather.get", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "<|tool_call>call:weather.get{location:<|\"|>London<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 19323, 236761, 828, 236782, 7125, 236787, 52, 27822, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "weather.get", "arguments": {"location": "London"}}]}, "tags": ["single-call", "x-name-punctuation"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/vllm-hyphenated-function-name expected_match

Checks that did not pass for gemma4/vllm-hyphenated-function-name
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'-weather{location:<\|"\|>London<\|"\|>}\'\nTransformed: \'-weather{"location":"London"}...
`expected_match`	`token`	fail	parser raised 'ValueError: json: could not parse after dialect transforms.\nOriginal: \'-weather{location:<\|"\|>London<\|"\|>}\'\nTransformed: \'-weather{"location":"London"}...

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-hyphenated-function-name --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 8 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "ValueError: json: could not parse after dialect transforms.\nOriginal: '-weather{location:<|\"|>London<|\"|>}'\nTransformed: '-weather{\"location\":\"London\"}'\nError: Expecting value: line 1 column 1 (char 0)",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "London"
-      },
-      "name": "get-weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|tool_call>call:get-weather{location:<|"|>London<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "London"
      },
      "name": "get-weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-hyphenated-function-name", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L392", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get-weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "<|tool_call>call:get-weather{location:<|\"|>London<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 828, 236772, 19323, 236782, 7125, 236787, 52, 27822, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get-weather", "arguments": {"location": "London"}}]}, "tags": ["single-call", "x-name-punctuation"], "notes": "OpenAI tool names may contain '-'; the template renders them verbatim after 'call:'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

fail gemma4/vllm-malformed-no-brace expected_error

Checks that did not pass for gemma4/vllm-malformed-no-brace
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	outcome exception "ValueError: json parser could not parse region as JSON.\nContent: ' no brace'\nError: Expecting value: line 1 column 2 (char 1)"; accepted: ['content_passthrough', 'no_tool_calls']
`expected_error`	`token`	fail	outcome exception "ValueError: json parser could not parse region as JSON.\nContent: ' no brace'\nError: Expecting value: line 1 column 2 (char 1)"; accepted: ['content_passthrough', 'no_tool_calls']

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/malformed.jsonl --id gemma4/vllm-malformed-no-brace --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 1 of fixtures/gemma4/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "ValueError: json parser could not parse region as JSON.\nContent: ' no brace'\nError: Expecting value: line 1 column 2 (char 1)",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942. Tags: malformed.

Expected graceful failure: The call has no '{...}' argument object and names no offered tool; it is not a valid call. (accept: no_tool_calls, content_passthrough).

Raw output

<|tool_call>call:bad_func no brace<tool_call|>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-malformed-no-brace", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L942", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "<|tool_call>call:bad_func no brace<tool_call|>", "output_token_ids": [48, 6639, 236787, 15242, 236779, 6823, 951, 24088, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "The call has no '{...}' argument object and names no offered tool; it is not a valid call.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed"], "notes": "vLLM's test only asserts that the recovered name stays bounded by <tool_call|> (it returns a call named 'bad_func no brace'); per the format there is no valid call here. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/empty-arguments-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/empty-arguments-thinking
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'get_time takes no arguments.', got 'get_time takes no arguments.\n'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream 'get_time takes no arguments.', got 'get_time takes no arguments.\n'

Synthetic strategies (not counted): expected_match@one soft pass, expected_match@special soft pass, expected_match@rand:1:8 soft pass, expected_match@rand:2:8 soft pass, expected_match@rand:3:8 soft pass, expected_match@rand:4:8 soft pass, expected_match@rand:5:8 soft pass, stream_equals_nonstream@one soft pass, stream_equals_nonstream@special soft pass, stream_equals_nonstream@rand:1:8 soft pass, stream_equals_nonstream@rand:2:8 soft pass, stream_equals_nonstream@rand:3:8 soft pass, stream_equals_nonstream@rand:4:8 soft pass, stream_equals_nonstream@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/basic.jsonl --id gemma4/empty-arguments-thinking --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 8 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "get_time takes no arguments.",
+  "reasoning_content": "get_time takes no arguments.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, empty-arguments, reasoning.

Raw output

<|channel>thought
get_time takes no arguments.
<channel|><|tool_call>call:get_time{}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "get_time takes no arguments.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/empty-arguments-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<|channel>thought\nget_time takes no arguments.\n<channel|><|tool_call>call:get_time{}<tool_call|>", "output_token_ids": [100, 45518, 107, 828, 236779, 2289, 4716, 951, 12130, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 2289, 16454, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "get_time takes no arguments.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": true,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/no-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/no-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream 'The user is just greeting me. No tool is needed.', got 'The user is just greeting me. No tool is needed.\n'

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/basic.jsonl --id gemma4/no-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 4 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Hello! How can I help you today?",
-  "reasoning_content": "The user is just greeting me. No tool is needed.",
+  "reasoning_content": "The user is just greeting me. No tool is needed.\n",
   "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: no-call, reasoning.

Raw output

<|channel>thought
The user is just greeting me. No tool is needed.
<channel|>Hello! How can I help you today?

Expected parse

{
  "content": "Hello! How can I help you today?",
  "reasoning_content": "The user is just greeting me. No tool is needed.",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/no-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nThe user is just greeting me. No tool is needed.\n<channel|>Hello! How can I help you today?", "output_token_ids": [100, 45518, 107, 818, 2430, 563, 1164, 26227, 786, 236761, 2301, 5904, 563, 4354, 236761, 107, 101, 9259, 236888, 2088, 740, 564, 1601, 611, 3124, 236881], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": "Hello! How can I help you today?", "reasoning_content": "The user is just greeting me. No tool is needed.", "tool_calls": []}, "tags": ["no-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": true,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/parallel-two-calls-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/parallel-two-calls-thinking
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I should call the tools.', got 'I should call the tools.\n'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream 'I should call the tools.', got 'I should call the tools.\n'

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/parallel.jsonl --id gemma4/parallel-two-calls-thinking --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 1 of fixtures/gemma4/parallel.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "I should call the tools.",
+  "reasoning_content": "I should call the tools.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: parallel-calls, reasoning, unicode, nested-json, string-escapes.

Raw output

<|channel>thought
I should call the tools.
<channel|><|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:<|"|>café "best"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "I should call the tools.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/parallel-two-calls-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|channel>thought\nI should call the tools.\n<channel|><|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:<|\"|>café \"best\"<|\"|>}<tool_call|>", "output_token_ids": [100, 45518, 107, 236777, 1374, 2246, 506, 6436, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787, 52, 123125, 236859, 623, 9783, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "I should call the tools.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "unicode", "nested-json", "string-escapes"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": true,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/sglang-text-before-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/sglang-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	soft pass	content (whitespace only): expected 'Some text before ', got 'Some text before'
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Some text before', got 'Some text before '

Synthetic strategies (not counted): stream_equals_nonstream@one soft pass, stream_equals_nonstream@special soft pass, stream_equals_nonstream@rand:1:8 soft pass, stream_equals_nonstream@rand:2:8 soft pass, stream_equals_nonstream@rand:3:8 soft pass, stream_equals_nonstream@rand:4:8 soft pass, stream_equals_nonstream@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 13 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before ",
+  "content": "Some text before",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "Some text before ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|>

Expected parse

{
  "content": "Some text before ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/sglang-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5535", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|>", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/single-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/single-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream "The user wants the current weather in Paris. I'll call get_weather.", got "The user wants the current weather in Paris. I'll call get_weather.\n"

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/basic.jsonl --id gemma4/single-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 2 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
+  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, reasoning.

Raw output

<|channel>thought
The user wants the current weather in Paris. I'll call get_weather.
<channel|><|tool_call>call:get_weather{city:<|"|>Paris<|"|>,unit:<|"|>c<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/single-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nThe user wants the current weather in Paris. I'll call get_weather.\n<channel|><|tool_call>call:get_weather{city:<|\"|>Paris<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|>", "output_token_ids": [100, 45518, 107, 818, 2430, 8150, 506, 1873, 7606, 528, 9079, 236761, 564, 236789, 859, 2246, 974, 236779, 19323, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": true,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/text-after-call-thinking expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/text-after-call-thinking
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	reasoning_content (whitespace only): expected 'I need live weather data for London.', got 'I need live weather data for London.\n'
`stream_equals_nonstream`	`token`	soft pass	reasoning_content (whitespace only): nonstream 'I need live weather data for London.', got 'I need live weather data for London.\n'

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/basic.jsonl --id gemma4/text-after-call-thinking --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 6 of fixtures/gemma4/basic.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,5 +1,5 @@
 {
   "content": "Let me check the current conditions in London.",
-  "reasoning_content": "I need live weather data for London.",
+  "reasoning_content": "I need live weather data for London.\n",
   "tool_calls": [
     {

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, text-after-call, reasoning.

Raw output

<|channel>thought
I need live weather data for London.
<channel|><|tool_call>call:get_weather{city:<|"|>London<|"|>}<tool_call|>Let me check the current conditions in London.

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": "I need live weather data for London.",
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/text-after-call-thinking", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|channel>thought\nI need live weather data for London.\n<channel|><|tool_call>call:get_weather{city:<|\"|>London<|\"|>}<tool_call|>Let me check the current conditions in London.", "output_token_ids": [100, 45518, 107, 236777, 1202, 3892, 7606, 1262, 573, 5860, 236761, 107, 101, 48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 27822, 52, 236783, 49, 6481, 786, 2426, 506, 1873, 3439, 528, 5860, 236761], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n", "thinking": true, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": "I need live weather data for London.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-after-call", "reasoning"], "notes": "History render of the official template; generation stops on <|tool_response>/<turn|>."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": true,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

soft pass gemma4/vllm-text-before-call expected_match, stream_equals_nonstream

Checks that did not pass for gemma4/vllm-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`token`	soft pass	content (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
`stream_equals_nonstream`	`token`	soft pass	content (whitespace only): nonstream 'Let me check the weather for you.', got 'Let me check the weather for you. '

Minimal repro

uv run canitoolcall run --engine transformers --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/transformers.sh; this run used transformers 5.17.0. The fixture is line 4 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

Matches the expected parse.

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you. ",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Raw output

Let me check the weather for you. <|tool_call>call:get_weather{location:<|"|>Paris<|"|>}<tool_call|>

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gemma4/vllm-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Let me check the weather for you. <|tool_call>call:get_weather{location:<|\"|>Paris<|\"|>}<tool_call|>", "output_token_ids": [6481, 786, 2426, 506, 7606, 573, 611, 236761, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 50429, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "text-before-call"], "notes": "The template renders text after calls, but models also emit text before them. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "chat_template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4",
  "nonstream_entrypoint": "tokenizer.parse_response(output_ids, prefix=prompt, tools=tools)",
  "prompt": {
    "add_generation_prompt": true,
    "enable_thinking": false,
    "user_message": "Please help with the request, using the available tools if needed."
  },
  "reasoning_parser": "response_template",
  "response_template_sha256": "6c8545de7fd056896d82142383de94d98339d44945fe00285f0c4c4ba1ed3a44",
  "response_template_source": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475:tokenizer_config.json",
  "stop_token_appended": null,
  "stream_detokenizer": "tokenizers.decoders.DecodeStream(skip_special_tokens=False), one id per step",
  "stream_entrypoint": "ResponseParser(response_template, prefix=prompt, tools=tools).feed/.finalize",
  "stream_events": "transformers.cli.serving.utils.response_events_to_chunks",
  "tokenizer": "google/gemma-4-31B-it@842da3794eaa0b77d5f08bae87a17459d91ff475",
  "tokenizer_mode": "hf",
  "tool_parser": "response_template",
  "tools_passed_to_parser": true
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model