CanIToolCall caniuse.com for tool calling

Gemma 4 (<|tool_call>call:NAME{...} object notation) on ollama 7af39318

fail 83% strict pass 40 pass · 1 soft pass · 7 fail

Run

Engine
ollama 7af39318 (7af393188defd52d370464de0d2064649cab9b41)
Run
to
Platform
linux-x86_64, Python 3.12.3, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "go_version": "go1.26.0",
  "llama_cpp_build": "b11081",
  "llama_cpp_commit": "161755f",
  "pinned_version": "7af393188defd52d370464de0d2064649cab9b41",
  "runner": "llama-server token rendering (preserved_tokens) + model/parsers via server/routes.go flow",
  "tokens_per_step": "one"
}
Results file
data/ollama-7af39318.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 37 1 6 0 84%
expected_error 3 0 1 0 75%
stream_equals_nonstream 48 0 0 0 100%
no_leakage 47 0 1 0 98%
arguments_json 35 0 0 0 100%
arguments_schema 35 0 0 0 100%
parallel_order 4 0 0 0 100%

7 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail gemma4/bug-ollama-18390-key-with-spaces expected_match
Checks that did not pass for gemma4/bug-ollama-18390-key-with-spaces
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['create_workflow'], got []
expected_matchtoken failtool_calls: expected ['create_workflow'], got []

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/bug-reports.jsonl --id gemma4/bug-ollama-18390-key-with-spaces --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 1 of fixtures/gemma4/bug-reports.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,24 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "connections": {
-          "Basic LLM Chain": {
-            "main": [
-              [
-                {
-                  "index": 0,
-                  "node": "X",
-                  "type": "main"
-                }
-              ]
-            ]
-          }
-        },
-        "name": "Demo"
-      },
-      "name": "create_workflow"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: bug_report, https://github.com/ollama/ollama/issues/18390. Tags: single-call, nested-json, regression, x-key-with-space.

Raw output

<|tool_call>call:create_workflow{name:<|"|>Demo<|"|>, connections:{Basic LLM Chain:{main:[[{node:<|"|>X<|"|>, type:<|"|>main<|"|>, index:0}]]}}}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "connections": {
          "Basic LLM Chain": {
            "main": [
              [
                {
                  "index": 0,
                  "node": "X",
                  "type": "main"
                }
              ]
            ]
          }
        },
        "name": "Demo"
      },
      "name": "create_workflow"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/bug-ollama-18390-key-with-spaces", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ollama/ollama/issues/18390", "revision": "issue body (2026-09-11)", "license": "NOASSERTION"}, "tools": [{"type": "function", "function": {"name": "create_workflow", "parameters": {"type": "object", "properties": {"name": {"type": "string"}, "connections": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:create_workflow{name:<|\"|>Demo<|\"|>, connections:{Basic LLM Chain:{main:[[{node:<|\"|>X<|\"|>, type:<|\"|>main<|\"|>, index:0}]]}}}<tool_call|>", "output_token_ids": [48, 6639, 236787, 3663, 236779, 95778, 236782, 1201, 236787, 52, 33846, 52, 236764, 13792, 29616, 18397, 32050, 236792, 35211, 29616, 3241, 24845, 53103, 5335, 236787, 52, 236917, 52, 236764, 1722, 236787, 52, 3241, 52, 236764, 3546, 236787, 236771, 236783, 10660, 19465, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_workflow", "arguments": {"name": "Demo", "connections": {"Basic LLM Chain": {"main": [[{"node": "X", "type": "main", "index": 0}]]}}}}]}, "tags": ["single-call", "nested-json", "regression", "x-key-with-space"], "notes": "The 'bare key with spaces' input of the issue's parser-level Go test. Ollama dropped the whole call. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/edge-values-key-with-space expected_match
Checks that did not pass for gemma4/edge-values-key-with-space
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['record'], got []
expected_matchtoken failtool_calls: expected ['record'], got []

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/edge.jsonl --id gemma4/edge-values-key-with-space --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 2 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,21 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "flag": false,
-        "key with space": 1.5e-05,
-        "none": null,
-        "obj": {
-          "B": 1,
-          "a": [
-            1,
-            "t"
-          ]
-        },
-        "s": "x,y} \"q\""
-      },
-      "name": "record"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, numeric-arguments, string-escapes, nested-json, x-key-with-space, x-dictsort-order.

Raw output

<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|"|>t<|"|>],B:1},s:<|"|>x,y} "q"<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "flag": false,
        "key with space": 1.5e-05,
        "none": null,
        "obj": {
          "B": 1,
          "a": [
            1,
            "t"
          ]
        },
        "s": "x,y} \"q\""
      },
      "name": "record"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/edge-values-key-with-space", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "record", "description": "Store a record with arbitrary keys.", "parameters": {"type": "object", "properties": {"key with space": {"type": "number"}, "s": {"type": "string"}, "flag": {"type": "boolean"}, "none": {"anyOf": [{"type": "string"}, {"type": "null"}], "nullable": true}, "obj": {"type": "object"}}}}}], "raw_output": "<|tool_call>call:record{flag:false,key with space:1.5e-05,none:null,obj:{a:[1,<|\"|>t<|\"|>],B:1},s:<|\"|>x,y} \"q\"<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17789, 236782, 16213, 236787, 4530, 236764, 2478, 607, 2557, 236787, 236770, 236761, 236810, 236744, 236772, 236771, 236810, 236764, 9745, 236787, 6455, 236764, 7837, 29616, 236746, 24845, 236770, 236764, 52, 236745, 52, 1604, 236799, 236787, 236770, 1263, 236751, 236787, 52, 236781, 236764, 236762, 236783, 623, 236809, 236775, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "record", "arguments": {"key with space": 1.5e-05, "s": "x,y} \"q\"", "flag": false, "none": null, "obj": {"B": 1, "a": [1, "t"]}}}]}, "tags": ["single-call", "numeric-arguments", "string-escapes", "nested-json", "x-key-with-space", "x-dictsort-order"], "notes": "Same input as the edge-value render in docs/formats/gemma4.md: a key containing spaces (unquoted), a Python-formatted float (1.5e-05), ',', '}' and '\"' inside a string, and case-insensitive key order. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/many-strings-then-string-array expected_match
Checks that did not pass for gemma4/many-strings-then-string-array
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['submit'], got []
expected_matchtoken failtool_calls: expected ['submit'], got []

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/edge.jsonl --id gemma4/many-strings-then-string-array --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 12 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,59 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "k00": "v00",
-        "k01": "v01",
-        "k02": "v02",
-        "k03": "v03",
-        "k04": "v04",
-        "k05": "v05",
-        "k06": "v06",
-        "k07": "v07",
-        "k08": "v08",
-        "k09": "v09",
-        "k10": "v10",
-        "k11": "v11",
-        "k12": "v12",
-        "k13": "v13",
-        "k14": "v14",
-        "k15": "v15",
-        "k16": "v16",
-        "k17": "v17",
-        "k18": "v18",
-        "k19": "v19",
-        "k20": "v20",
-        "k21": "v21",
-        "k22": "v22",
-        "k23": "v23",
-        "k24": "v24",
-        "k25": "v25",
-        "k26": "v26",
-        "k27": "v27",
-        "k28": "v28",
-        "k29": "v29",
-        "k30": "v30",
-        "k31": "v31",
-        "k32": "v32",
-        "k33": "v33",
-        "k34": "v34",
-        "k35": "v35",
-        "k36": "v36",
-        "k37": "v37",
-        "k38": "v38",
-        "k39": "v39",
-        "k40": "v40",
-        "k41": "v41",
-        "k42": "v42",
-        "k43": "v43",
-        "k44": "v44",
-        "zz_members": [
-          "a",
-          "b"
-        ]
-      },
-      "name": "submit"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, long-arguments, regression.

Raw output

<|tool_call>call:submit{k00:<|"|>v00<|"|>,k01:<|"|>v01<|"|>,k02:<|"|>v02<|"|>,k03:<|"|>v03<|"|>,k04:<|"|>v04<|"|>,k05:<|"|>v05<|"|>,k06:<|"|>v06<|"|>,k07:<|"|>v07<|"|>,k08:<|"|>v08<|"|>,k09:<|"|>v09<|"|>,k10:<|"|>v10<|"|>,k11:<|"|>v11<|"|>,k12:<|"|>v12<|"|>,k13:<|"|>v13<|"|>,k14:<|"|>v14<|"|>,k15:<|"|>v15<|"|>,k16:<|"|>v16<|"|>,k17:<|"|>v17<|"|>,k18:<|"|>v18<|"|>,k19:<|"|>v19<|"|>,k20:<|"|>v20<|"|>,k21:<|"|>v21<|"|>,k22:<|"|>v22<|"|>,k23:<|"|>v23<|"|>,k24:<|"|>v24<|"|>,k25:<|"|>v25<|"|>,k26:<|"|>v26<|"|>,k27:<|"|>v27<|"|>,k28:<|"|>v28<|"|>,k29:<|"|>v29<|"|>,k30:<|"|>v30<|"|>,k31:<|"|>v31<|"|>,k32:<|"|>v32<|"|>,k33:<|"|>v33<|"|>,k34:<|"|>v34<|"|>,k35:<|"|>v35<|"|>,k36:<|"|>v36<|"|>,k37:<|"|>v37<|"|>,k38:<|"|>v38<|"|>,k39:<|"|>v39<|"|>,k40:<|"|>v40<|"|>,k41:<|"|>v41<|"|>,k42:<|"|>v42<|"|>,k43:<|"|>v43<|"|>,k44:<|"|>v44<|"|>,zz_members:[<|"|>a<|"|>,<|"|>b<|"|>]}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "k00": "v00",
        "k01": "v01",
        "k02": "v02",
        "k03": "v03",
        "k04": "v04",
        "k05": "v05",
        "k06": "v06",
        "k07": "v07",
        "k08": "v08",
        "k09": "v09",
        "k10": "v10",
        "k11": "v11",
        "k12": "v12",
        "k13": "v13",
        "k14": "v14",
        "k15": "v15",
        "k16": "v16",
        "k17": "v17",
        "k18": "v18",
        "k19": "v19",
        "k20": "v20",
        "k21": "v21",
        "k22": "v22",
        "k23": "v23",
        "k24": "v24",
        "k25": "v25",
        "k26": "v26",
        "k27": "v27",
        "k28": "v28",
        "k29": "v29",
        "k30": "v30",
        "k31": "v31",
        "k32": "v32",
        "k33": "v33",
        "k34": "v34",
        "k35": "v35",
        "k36": "v36",
        "k37": "v37",
        "k38": "v38",
        "k39": "v39",
        "k40": "v40",
        "k41": "v41",
        "k42": "v42",
        "k43": "v43",
        "k44": "v44",
        "zz_members": [
          "a",
          "b"
        ]
      },
      "name": "submit"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/many-strings-then-string-array", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "submit", "description": "Submit all provided fields.", "parameters": {"type": "object", "properties": {"k00": {"type": "string"}, "k01": {"type": "string"}, "k02": {"type": "string"}, "k03": {"type": "string"}, "k04": {"type": "string"}, "k05": {"type": "string"}, "k06": {"type": "string"}, "k07": {"type": "string"}, "k08": {"type": "string"}, "k09": {"type": "string"}, "k10": {"type": "string"}, "k11": {"type": "string"}, "k12": {"type": "string"}, "k13": {"type": "string"}, "k14": {"type": "string"}, "k15": {"type": "string"}, "k16": {"type": "string"}, "k17": {"type": "string"}, "k18": {"type": "string"}, "k19": {"type": "string"}, "k20": {"type": "string"}, "k21": {"type": "string"}, "k22": {"type": "string"}, "k23": {"type": "string"}, "k24": {"type": "string"}, "k25": {"type": "string"}, "k26": {"type": "string"}, "k27": {"type": "string"}, "k28": {"type": "string"}, "k29": {"type": "string"}, "k30": {"type": "string"}, "k31": {"type": "string"}, "k32": {"type": "string"}, "k33": {"type": "string"}, "k34": {"type": "string"}, "k35": {"type": "string"}, "k36": {"type": "string"}, "k37": {"type": "string"}, "k38": {"type": "string"}, "k39": {"type": "string"}, "k40": {"type": "string"}, "k41": {"type": "string"}, "k42": {"type": "string"}, "k43": {"type": "string"}, "k44": {"type": "string"}, "zz_members": {"type": "array", "items": {"type": "string"}}}, "required": ["k00", "k01", "k02", "k03", "k04", "k05", "k06", "k07", "k08", "k09", "k10", "k11", "k12", "k13", "k14", "k15", "k16", "k17", "k18", "k19", "k20", "k21", "k22", "k23", "k24", "k25", "k26", "k27", "k28", "k29", "k30", "k31", "k32", "k33", "k34", "k35", "k36", "k37", "k38", "k39", "k40", "k41", "k42", "k43", "k44", "zz_members"]}}}], "raw_output": "<|tool_call>call:submit{k00:<|\"|>v00<|\"|>,k01:<|\"|>v01<|\"|>,k02:<|\"|>v02<|\"|>,k03:<|\"|>v03<|\"|>,k04:<|\"|>v04<|\"|>,k05:<|\"|>v05<|\"|>,k06:<|\"|>v06<|\"|>,k07:<|\"|>v07<|\"|>,k08:<|\"|>v08<|\"|>,k09:<|\"|>v09<|\"|>,k10:<|\"|>v10<|\"|>,k11:<|\"|>v11<|\"|>,k12:<|\"|>v12<|\"|>,k13:<|\"|>v13<|\"|>,k14:<|\"|>v14<|\"|>,k15:<|\"|>v15<|\"|>,k16:<|\"|>v16<|\"|>,k17:<|\"|>v17<|\"|>,k18:<|\"|>v18<|\"|>,k19:<|\"|>v19<|\"|>,k20:<|\"|>v20<|\"|>,k21:<|\"|>v21<|\"|>,k22:<|\"|>v22<|\"|>,k23:<|\"|>v23<|\"|>,k24:<|\"|>v24<|\"|>,k25:<|\"|>v25<|\"|>,k26:<|\"|>v26<|\"|>,k27:<|\"|>v27<|\"|>,k28:<|\"|>v28<|\"|>,k29:<|\"|>v29<|\"|>,k30:<|\"|>v30<|\"|>,k31:<|\"|>v31<|\"|>,k32:<|\"|>v32<|\"|>,k33:<|\"|>v33<|\"|>,k34:<|\"|>v34<|\"|>,k35:<|\"|>v35<|\"|>,k36:<|\"|>v36<|\"|>,k37:<|\"|>v37<|\"|>,k38:<|\"|>v38<|\"|>,k39:<|\"|>v39<|\"|>,k40:<|\"|>v40<|\"|>,k41:<|\"|>v41<|\"|>,k42:<|\"|>v42<|\"|>,k43:<|\"|>v43<|\"|>,k44:<|\"|>v44<|\"|>,zz_members:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]}<tool_call|>", "output_token_ids": [48, 6639, 236787, 15037, 236782, 236767, 236771, 236771, 236787, 52, 236766, 236771, 236771, 52, 236764, 236767, 236771, 236770, 236787, 52, 236766, 236771, 236770, 52, 236764, 236767, 236771, 236778, 236787, 52, 236766, 236771, 236778, 52, 236764, 236767, 236771, 236800, 236787, 52, 236766, 236771, 236800, 52, 236764, 236767, 236771, 236812, 236787, 52, 236766, 236771, 236812, 52, 236764, 236767, 236771, 236810, 236787, 52, 236766, 236771, 236810, 52, 236764, 236767, 236771, 236825, 236787, 52, 236766, 236771, 236825, 52, 236764, 236767, 236771, 236832, 236787, 52, 236766, 236771, 236832, 52, 236764, 236767, 236771, 236828, 236787, 52, 236766, 236771, 236828, 52, 236764, 236767, 236771, 236819, 236787, 52, 236766, 236771, 236819, 52, 236764, 236767, 236770, 236771, 236787, 52, 236766, 236770, 236771, 52, 236764, 236767, 236770, 236770, 236787, 52, 236766, 236770, 236770, 52, 236764, 236767, 236770, 236778, 236787, 52, 236766, 236770, 236778, 52, 236764, 236767, 236770, 236800, 236787, 52, 236766, 236770, 236800, 52, 236764, 236767, 236770, 236812, 236787, 52, 236766, 236770, 236812, 52, 236764, 236767, 236770, 236810, 236787, 52, 236766, 236770, 236810, 52, 236764, 236767, 236770, 236825, 236787, 52, 236766, 236770, 236825, 52, 236764, 236767, 236770, 236832, 236787, 52, 236766, 236770, 236832, 52, 236764, 236767, 236770, 236828, 236787, 52, 236766, 236770, 236828, 52, 236764, 236767, 236770, 236819, 236787, 52, 236766, 236770, 236819, 52, 236764, 236767, 236778, 236771, 236787, 52, 236766, 236778, 236771, 52, 236764, 236767, 236778, 236770, 236787, 52, 236766, 236778, 236770, 52, 236764, 236767, 236778, 236778, 236787, 52, 236766, 236778, 236778, 52, 236764, 236767, 236778, 236800, 236787, 52, 236766, 236778, 236800, 52, 236764, 236767, 236778, 236812, 236787, 52, 236766, 236778, 236812, 52, 236764, 236767, 236778, 236810, 236787, 52, 236766, 236778, 236810, 52, 236764, 236767, 236778, 236825, 236787, 52, 236766, 236778, 236825, 52, 236764, 236767, 236778, 236832, 236787, 52, 236766, 236778, 236832, 52, 236764, 236767, 236778, 236828, 236787, 52, 236766, 236778, 236828, 52, 236764, 236767, 236778, 236819, 236787, 52, 236766, 236778, 236819, 52, 236764, 236767, 236800, 236771, 236787, 52, 236766, 236800, 236771, 52, 236764, 236767, 236800, 236770, 236787, 52, 236766, 236800, 236770, 52, 236764, 236767, 236800, 236778, 236787, 52, 236766, 236800, 236778, 52, 236764, 236767, 236800, 236800, 236787, 52, 236766, 236800, 236800, 52, 236764, 236767, 236800, 236812, 236787, 52, 236766, 236800, 236812, 52, 236764, 236767, 236800, 236810, 236787, 52, 236766, 236800, 236810, 52, 236764, 236767, 236800, 236825, 236787, 52, 236766, 236800, 236825, 52, 236764, 236767, 236800, 236832, 236787, 52, 236766, 236800, 236832, 52, 236764, 236767, 236800, 236828, 236787, 52, 236766, 236800, 236828, 52, 236764, 236767, 236800, 236819, 236787, 52, 236766, 236800, 236819, 52, 236764, 236767, 236812, 236771, 236787, 52, 236766, 236812, 236771, 52, 236764, 236767, 236812, 236770, 236787, 52, 236766, 236812, 236770, 52, 236764, 236767, 236812, 236778, 236787, 52, 236766, 236812, 236778, 52, 236764, 236767, 236812, 236800, 236787, 52, 236766, 236812, 236800, 52, 236764, 236767, 236812, 236812, 236787, 52, 236766, 236812, 236812, 52, 236764, 9961, 236779, 29552, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 21111, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "submit", "arguments": {"k00": "v00", "k01": "v01", "k02": "v02", "k03": "v03", "k04": "v04", "k05": "v05", "k06": "v06", "k07": "v07", "k08": "v08", "k09": "v09", "k10": "v10", "k11": "v11", "k12": "v12", "k13": "v13", "k14": "v14", "k15": "v15", "k16": "v16", "k17": "v17", "k18": "v18", "k19": "v19", "k20": "v20", "k21": "v21", "k22": "v22", "k23": "v23", "k24": "v24", "k25": "v25", "k26": "v26", "k27": "v27", "k28": "v28", "k29": "v29", "k30": "v30", "k31": "v31", "k32": "v32", "k33": "v33", "k34": "v34", "k35": "v35", "k36": "v36", "k37": "v37", "k38": "v38", "k39": "v39", "k40": "v40", "k41": "v41", "k42": "v42", "k43": "v43", "k44": "v44", "zz_members": ["a", "b"]}}]}, "tags": ["single-call", "long-arguments", "regression"], "notes": "45 string values followed by a two-string array: the exact input of https://github.com/ollama/ollama/issues/18354 (string-placeholder collision dropped the call). History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/marker-in-arguments expected_match, no_leakage
Checks that did not pass for gemma4/marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got 'and close thoughts with <channel|>.<|"|>}<tool_call|>'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text":"Wrap calls as \\u003c|tool_call\\u003ecall:f{"}'
expected_matchtoken failcontent: expected None, got 'and close thoughts with <channel|>.<|"|>}<tool_call|>'; tool_calls[0].arguments: expected '{"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}', got '{"text":"Wrap calls as \\u003c|tool_call\\u003ecall:f{"}'
no_leakagenonstream failcontent contains '<tool_call|>'; content contains '<|"|>'; content contains '<channel|>'
no_leakagetoken failcontent contains '<tool_call|>'; content contains '<|"|>'; content contains '<channel|>'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, no_leakage@one fail, no_leakage@special fail, no_leakage@rand:1:8 fail, no_leakage@rand:2:8 fail, no_leakage@rand:3:8 fail, no_leakage@rand:4:8 fail, no_leakage@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/edge.jsonl --id gemma4/marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 4 of fixtures/gemma4/edge.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": "and close thoughts with <channel|>.<|\"|>}<tool_call|>",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
+        "text": "Wrap calls as <|tool_call>call:f{"
       },
       "name": "echo"

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: single-call, marker-in-arguments.

Raw output

<|tool_call>call:echo{text:<|"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|"|>}<tool_call|>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/marker-in-arguments", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<|tool_call>call:echo{text:<|\"|>Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>.<|\"|>}<tool_call|>", "output_token_ids": [48, 6639, 236787, 17454, 236782, 1005, 236787, 52, 47352, 9139, 618, 236743, 48, 6639, 236787, 236760, 16454, 49, 532, 3107, 12018, 607, 236743, 101, 236761, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Wrap calls as <|tool_call>call:f{}<tool_call|> and close thoughts with <channel|>."}}]}, "tags": ["single-call", "marker-in-arguments"], "notes": "The string contains the text of the call/channel markers, which the official tokenizer maps to their special-token ids. Inside a <|\"|>-delimited string they are literal text. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/sglang-text-around-call expected_match
Checks that did not pass for gemma4/sglang-text-around-call
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Some text before after', got 'Some text before after'
expected_matchtoken failcontent: expected 'Some text before after', got 'Some text before after'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-around-call --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 14 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, token, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before  after",
+  "content": "Some text before after",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Some text before  after",
+  "content": "Some text beforeafter",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546. Tags: single-call, text-before-call, text-after-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|> after

Expected parse

{
  "content": "Some text before  after",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-text-around-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5546", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|> after", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49, 1308], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before  after", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "text-after-call"], "notes": "The streamed chunks of the test, joined. Content is all text outside the call. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/truncated-mid-string expected_error
Checks that did not pass for gemma4/truncated-mid-string
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the tool call: no complete call exists.

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-mid-string --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 1 of fixtures/gemma4/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated.

Expected graceful failure: Output cut by max_tokens inside the tool call: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|tool_call>call:get_weather{city:<|"|>Paris
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-mid-string", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Paris", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 50429], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the tool call: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated"], "notes": "Token prefix of the full render, cut right after 'Par' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
fail gemma4/truncated-second-parallel-call expected_match
Checks that did not pass for gemma4/truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failtool_calls: expected ['get_weather'], got ['get_weather', 'search']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/truncated.jsonl --id gemma4/truncated-second-parallel-call --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 5 of fixtures/gemma4/truncated.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,17 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "filters": {
+          "max": 3,
+          "tags": [
+            "a",
+            "b"
+          ]
+        },
+        "query": ""
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja. Tags: truncated, parallel-calls, unicode.

Raw output

<|tool_call>call:get_weather{city:<|"|>Zürich<|"|>,unit:<|"|>c<|"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|"|>a<|"|>,<|"|>b<|"|>]},query:

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/truncated-second-parallel-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "license": "Apache-2.0", "generator": "scripts/fixtures/gemma4/build.py", "template_sha256": "ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<|tool_call>call:get_weather{city:<|\"|>Zürich<|\"|>,unit:<|\"|>c<|\"|>}<tool_call|><|tool_call>call:search{filters:{max:3,tags:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]},query:", "output_token_ids": [48, 6639, 236787, 828, 236779, 19323, 236782, 13319, 236787, 52, 236953, 105214, 52, 236764, 6805, 236787, 52, 236755, 52, 236783, 49, 48, 6639, 236787, 2305, 236782, 28881, 29616, 2074, 236787, 236800, 236764, 19112, 24845, 52, 236746, 52, 236764, 52, 236763, 52, 59117, 3278, 236787], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode"], "notes": "The first call is complete; the second is cut. A correct parser keeps the complete call only. Token prefix of the full render, cut right after 'query:' appears. History render of the official template; generation stops on <|tool_response>/<turn|>."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
soft pass gemma4/vllm-text-before-call expected_match
Checks that did not pass for gemma4/vllm-text-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the weather for you.', got 'Let me check the weather for you. '

Synthetic strategies (not counted): expected_match@rand:4:8 soft pass, stream_equals_nonstream@one soft pass, stream_equals_nonstream@special soft pass, stream_equals_nonstream@rand:1:8 soft pass, stream_equals_nonstream@rand:2:8 soft pass, stream_equals_nonstream@rand:3:8 soft pass, stream_equals_nonstream@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/vllm-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 4 of fixtures/gemma4/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, token, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you. ",
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323. Tags: single-call, text-before-call.

Raw output

Let me check the weather for you. <|tool_call>call:get_weather{location:<|"|>Paris<|"|>}<tool_call|>

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/vllm-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_gemma4_tool_parser.py#L323", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Let me check the weather for you. <|tool_call>call:get_weather{location:<|\"|>Paris<|\"|>}<tool_call|>", "output_token_ids": [6481, 786, 2426, 506, 7606, 573, 611, 236761, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 50429, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Let me check the weather for you.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "text-before-call"], "notes": "The template renders text after calls, but models also emit text before them. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}
pass gemma4/sglang-text-before-call

Synthetic strategies (not counted): expected_match@one soft pass, expected_match@special soft pass, expected_match@rand:1:8 soft pass, expected_match@rand:2:8 soft pass, expected_match@rand:5:8 soft pass, stream_equals_nonstream@one soft pass, stream_equals_nonstream@special soft pass, stream_equals_nonstream@rand:1:8 soft pass, stream_equals_nonstream@rand:2:8 soft pass, stream_equals_nonstream@rand:5:8 soft pass

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gemma4/engine-tests.jsonl --id gemma4/sglang-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 13 of fixtures/gemma4/engine-tests.jsonl.

The results file does not include the observed parses for this fixture.

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5535. Tags: single-call, text-before-call.

Raw output

Some text before <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|>

Expected parse

{
  "content": "Some text before ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gemma4/sglang-text-before-call", "family": "gemma4", "models": ["google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-12B-it"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L5535", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string"}}}}}], "raw_output": "Some text before <|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|>", "output_token_ids": [9401, 1816, 1680, 236743, 48, 6639, 236787, 828, 236779, 19323, 236782, 7125, 236787, 52, 89265, 52, 236783, 49], "tokenizer": {"repo": "google/gemma-4-31B-it", "revision": "842da3794eaa0b77d5f08bae87a17459d91ff475", "mode": "hf"}, "generation_prompt": "<|turn>model\n<|channel>thought\n<channel|>", "thinking": false, "expected": {"content": "Some text before ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "text-before-call"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "gemma4",
  "parser_source": "Ollama library model config (registry.ollama.ai, fetched 2026-09-25): gemma4",
  "preserved_tokens": [
    "<|channel>",
    "<channel|>",
    "<|tool_call>",
    "<tool_call|>",
    "<|tool_response>",
    "<|\"|>"
  ],
  "think_effective": false,
  "think_requested": false,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "google--gemma-4-31B-it.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "842da3794eaa0b77d5f08bae87a17459d91ff475",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "c45a756342ca921303a5689e6ea3e4c5133e6ebfe79d158d91616c3597803e31"
}