CanIToolCall caniuse.com for tool calling

Llama (3.1, 3.2, 3.3, 4) on vllm 0.30.0

fail 26% strict pass 9 pass · 25 fail

Run

Engine
vllm 0.30.0
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}
Results file
data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 9 0 23 0 28%
expected_error 0 0 2 0 0%
stream_equals_nonstream 9 0 25 0 26%
split_invariance 16 0 18 0 47%
no_leakage 34 0 0 0 100%
arguments_json 25 0 3 0 89%
arguments_schema 25 0 3 0 89%
parallel_order 2 0 0 0 100%

Fixtures needing attention

fail llama/l3-bug-empty-json-object-in-content expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-bug-empty-json-object-in-content
CheckStrategyResultDetail
expected_matchone failcontent: expected '{}', got None
expected_matchspecial failcontent: expected '{}', got None
expected_matchtoken failcontent: expected '{}', got None
expected_matchrand:1:8 failcontent: expected '{}', got None
expected_matchrand:2:8 failcontent: expected '{}', got None
expected_matchrand:3:8 failcontent: expected '{}', got None
expected_matchrand:4:8 failcontent: expected '{}', got None
expected_matchrand:5:8 failcontent: expected '{}', got None
stream_equals_nonstreamone failcontent: nonstream '{}', got None
stream_equals_nonstreamspecial failcontent: nonstream '{}', got None
stream_equals_nonstreamtoken failcontent: nonstream '{}', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{}', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{}', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{}', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{}', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{}', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-empty-json-object-in-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 16 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Strategies: nonstream

Matches the expected parse.

{
  "content": "{}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{}

Expected parse

{
  "content": "{}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-empty-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue: input '{}' -> content '' (the message is gone)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{}", "output_token_ids": [6390], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-bug-leading-json-object-in-content expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-bug-leading-json-object-in-content
CheckStrategyResultDetail
expected_matchone failcontent: expected '{"a": 1} is a dict', got None
expected_matchspecial failcontent: expected '{"a": 1} is a dict', got None
expected_matchtoken failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:1:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:2:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:3:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:4:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:5:8 failcontent: expected '{"a": 1} is a dict', got None
stream_equals_nonstreamone failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamspecial failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamtoken failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"a": 1} is a dict', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"a": 1} is a dict', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-leading-json-object-in-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 15 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"a\": 1} is a dict",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Strategies: nonstream

Matches the expected parse.

{
  "content": "{\"a\": 1} is a dict",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"a": 1} is a dict

Expected parse

{
  "content": "{\"a\": 1} is a dict",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-leading-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's reproduction (Llama-3.2-1B-Instruct, tools=[get_weather])."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{\"a\": 1} is a dict", "output_token_ids": [5018, 64, 794, 220, 16, 92, 374, 264, 6587], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"a\": 1} is a dict", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "Not a call (no name/parameters). SGLang's llama3 parser deleted the leading JSON and returned 'is a dict'. The issue's expectation: when no call is found the text must be unchanged."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-bug-whole-call-single-delta expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-bug-whole-call-single-delta
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-whole-call-single-delta --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 17 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/48294. Tags: single-call, regression, x-single-delta, x-arguments-key, x-llama-json.

Raw output

{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-whole-call-single-delta", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/48294", "revision": "issue opened 2026-07-11", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's minimal reproduction for llama3_json."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "regression", "x-single-delta", "x-arguments-key", "x-llama-json"], "notes": "vLLM's llama3_json streaming path emitted nothing when this whole message arrived in one delta (the 'one' chunking strategy replays exactly that)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-empty-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_time'], got []
expected_matchspecial failtool_calls: expected ['get_time'], got []
expected_matchtoken failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{}', got ''
stream_equals_nonstreamone failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamtoken failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{}', got ''
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_time'] | rand:1:8 (fail): tool_calls: one [], got ['get_time'] | rand:2:8 (fail): tool_calls: one [], got ['get_time'] | rand:3:8 (fail): tool_calls: one [], got ['get_time'] | rand:4:8 (fail): tool_calls: one [], got ['get_time'] | rand:5:8 (fail): tool_calls: one [], got ['get_time']
arguments_jsontoken fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:1:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schematoken fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, empty-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "get_time", "parameters": {}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-empty-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"get_time\", \"parameters\": {}}", "output_token_ids": [5018, 609, 794, 330, 456, 3084, 498, 330, 14105, 794, 314, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-json-answer-not-a-call expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-json-answer-not-a-call
CheckStrategyResultDetail
expected_matchone failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchspecial failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchtoken failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:1:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:2:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:3:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:4:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:5:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamone failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamspecial failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamtoken failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"capital": "Paris", "country": "France"}', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-json-answer-not-a-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Strategies: nonstream

Matches the expected parse.

{
  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"capital": "Paris", "country": "France"}

Expected parse

{
  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-json-answer-not-a-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"capital\": \"Paris\", \"country\": \"France\"}", "output_token_ids": [5018, 66163, 794, 330, 60704, 498, 330, 11389, 794, 330, 50100, 9388], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"capital\": \"Paris\", \"country\": \"France\"}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "A plain JSON answer: it has no name/parameters keys, so it is content, not a call (cf. https://github.com/sgl-project/sglang/issues/35562)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-long-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-long-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['write_file'], got []
expected_matchspecial failtool_calls: expected ['write_file'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['write_file'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['write_file'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['write_file'] | rand:1:8 (fail): tool_calls: one [], got ['write_file'] | rand:2:8 (fail): tool_calls: one [], got ['write_file'] | rand:3:8 (fail): tool_calls: one [], got ['write_file'] | rand:4:8 (fail): tool_calls: one [], got ['write_file'] | rand:5:8 (fail): tool_calls: one [], got ['write_file']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-long-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
-        "path": "fox.txt"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
        "path": "fox.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, long-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog."}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
        "path": "fox.txt"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-long-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"fox.txt\", \"content\": \"000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown fox jumps over the lazy dog.\\n003: The quick brown fox jumps over the lazy dog.\\n004: The quick brown fox jumps over the lazy dog.\\n005: The quick brown fox jumps over the lazy dog.\\n006: The quick brown fox jumps over the lazy dog.\\n007: The quick brown fox jumps over the lazy dog.\\n008: The quick brown fox jumps over the lazy dog.\\n009: The quick brown fox jumps over the lazy dog.\\n010: The quick brown fox jumps over the lazy dog.\\n011: The quick brown fox jumps over the lazy dog.\\n012: The quick brown fox jumps over the lazy dog.\\n013: The quick brown fox jumps over the lazy dog.\\n014: The quick brown fox jumps over the lazy dog.\\n015: The quick brown fox jumps over the lazy dog.\\n016: The quick brown fox jumps over the lazy dog.\\n017: The quick brown fox jumps over the lazy dog.\\n018: The quick brown fox jumps over the lazy dog.\\n019: The quick brown fox jumps over the lazy dog.\\n020: The quick brown fox jumps over the lazy dog.\\n021: The quick brown fox jumps over the lazy dog.\\n022: The quick brown fox jumps over the lazy dog.\\n023: The quick brown fox jumps over the lazy dog.\\n024: The quick brown fox jumps over the lazy dog.\\n025: The quick brown fox jumps over the lazy dog.\\n026: The quick brown fox jumps over the lazy dog.\\n027: The quick brown fox jumps over the lazy dog.\\n028: The quick brown fox jumps over the lazy dog.\\n029: The quick brown fox jumps over the lazy dog.\\n030: The quick brown fox jumps over the lazy dog.\\n031: The quick brown fox jumps over the lazy dog.\\n032: The quick brown fox jumps over the lazy dog.\\n033: The quick brown fox jumps over the lazy dog.\\n034: The quick brown fox jumps over the lazy dog.\\n035: The quick brown fox jumps over the lazy dog.\\n036: The quick brown fox jumps over the lazy dog.\\n037: The quick brown fox jumps over the lazy dog.\\n038: The quick brown fox jumps over the lazy dog.\\n039: The quick brown fox jumps over the lazy dog.\\n040: The quick brown fox jumps over the lazy dog.\\n041: The quick brown fox jumps over the lazy dog.\\n042: The quick brown fox jumps over the lazy dog.\\n043: The quick brown fox jumps over the lazy dog.\\n044: The quick brown fox jumps over the lazy dog.\\n045: The quick brown fox jumps over the lazy dog.\\n046: The quick brown fox jumps over the lazy dog.\\n047: The quick brown fox jumps over the lazy dog.\\n048: The quick brown fox jumps over the lazy dog.\\n049: The quick brown fox jumps over the lazy dog.\\n050: The quick brown fox jumps over the lazy dog.\\n051: The quick brown fox jumps over the lazy dog.\\n052: The quick brown fox jumps over the lazy dog.\\n053: The quick brown fox jumps over the lazy dog.\\n054: The quick brown fox jumps over the lazy dog.\\n055: The quick brown fox jumps over the lazy dog.\\n056: The quick brown fox jumps over the lazy dog.\\n057: The quick brown fox jumps over the lazy dog.\\n058: The quick brown fox jumps over the lazy dog.\\n059: The quick brown fox jumps over the lazy dog.\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 15361, 3996, 498, 330, 1834, 794, 330, 931, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 4119, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 6726, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 6268, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 8759, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 8504, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11030, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11194, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11436, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 13858, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 7755, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 10731, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11531, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16368, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 15901, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16037, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 15794, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 17248, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16745, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18089, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11139, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11592, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18642, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20063, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 19592, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18070, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21641, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21360, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22000, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23273, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 14649, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18887, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21040, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 13103, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22379, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22407, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23110, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23587, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24462, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21602, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 12505, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20945, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22349, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 17776, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20078, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23785, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24222, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24970, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23904, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25307, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16193, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23545, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24130, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25210, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25230, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22913, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25921, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 26866, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24824, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 27033, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 1210, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog."}}]}, "tags": ["single-call", "long-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchone failtool_calls: expected ['write_file'], got []
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchtoken failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
stream_equals_nonstreamone failtool_calls: nonstream ['write_file'], got []
split_invariance* failvs one: special (fail): tool_calls: one [], got ['write_file'] | token (fail): tool_calls: one [], got ['write_file'] | rand:1:8 (fail): tool_calls: one [], got ['write_file'] | rand:2:8 (fail): tool_calls: one [], got ['write_file'] | rand:3:8 (fail): tool_calls: one [], got ['write_file'] | rand:4:8 (fail): tool_calls: one [], got ['write_file'] | rand:5:8 (fail): tool_calls: one [], got ['write_file']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
+        "content": "Tool calls may start with ; headers look like ipython.",
         "path": "docs/llama.md"
       },

Strategies: one

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
-        "path": "docs/llama.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, marker-in-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
        "path": "docs/llama.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-marker-in-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"docs/llama.md\", \"content\": \"Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 14452, 14, 657, 3105, 22030, 498, 330, 1834, 794, 330, 7896, 6880, 1253, 1212, 449, 220, 128010, 26, 7247, 1427, 1093, 220, 128006, 23799, 4690, 128007, 1210, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}]}, "tags": ["single-call", "marker-in-arguments", "x-parameters-key", "x-llama-json"], "notes": "The JSON string contains <|python_tag|> and header tokens, tokenized as the special tokens. A parser that splits on <|python_tag|> must not cut inside the string."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-meta-python-tag-type-function expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-meta-python-tag-type-function
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['trending_songs'], got []
expected_matchspecial failtool_calls: expected ['trending_songs'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['trending_songs'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['trending_songs'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['trending_songs'] | rand:1:8 (fail): tool_calls: one [], got ['trending_songs'] | rand:2:8 (fail): tool_calls: one [], got ['trending_songs'] | rand:3:8 (fail): tool_calls: one [], got ['trending_songs'] | rand:4:8 (fail): tool_calls: one [], got ['trending_songs'] | rand:5:8 (fail): tool_calls: one [], got ['trending_songs']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/recorded.jsonl --id llama/l3-meta-python-tag-type-function --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/llama/recorded.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "genre": "all",
-        "n": "10"
-      },
-      "name": "trending_songs"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "genre": "all",
        "n": "10"
      },
      "name": "trending_songs"
    }
  ]
}

Fixture

Provenance: spec_example, https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama3_3/prompt_format.md#L403-L412. Tags: single-call, x-llama-json, x-python-tag-prefix, x-extra-type-key, x-eom-terminator.

Raw output

<|python_tag|>{
    "type": "function",
    "name": "trending_songs",
    "parameters": {
        "n": "10",
        "genre": "all"
    }
}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "genre": "all",
        "n": "10"
      },
      "name": "trending_songs"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-meta-python-tag-type-function", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "spec_example", "source_url": "https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama3_3/prompt_format.md#L403-L412", "revision": "0e0b8c519242d5833d8c11bffc1232b77ad7f301", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "attribution": "Llama 3.3 is licensed under the Llama 3.3 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. Short example quoted with attribution (THIRD_PARTY_NOTICES.md)."}, "tools": [{"type": "function", "function": {"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"type": "object", "properties": {"n": {"description": "The number of songs to return"}, "genre": {"description": "The genre of the songs to return"}}, "required": ["n"]}}}], "raw_output": "<|python_tag|>{\n    \"type\": \"function\",\n    \"name\": \"trending_songs\",\n    \"parameters\": {\n        \"n\": \"10\",\n        \"genre\": \"all\"\n    }\n}", "output_token_ids": [128010, 517, 262, 330, 1337, 794, 330, 1723, 761, 262, 330, 609, 794, 330, 83, 63094, 646, 13483, 761, 262, 330, 14105, 794, 341, 286, 330, 77, 794, 330, 605, 761, 286, 330, 34713, 794, 330, 543, 702, 262, 457, 92], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "trending_songs", "arguments": {"n": "10", "genre": "all"}}]}, "tags": ["single-call", "x-llama-json", "x-python-tag-prefix", "x-extra-type-key", "x-eom-terminator"], "notes": "Quoted verbatim from the 'Model Response Format' block of Meta's prompt-format doc (spec_example: official format documentation, not a generation captured by this project). The stop token that ends the block is cut, as spec/README.md requires. The model prefixed <|python_tag|>, added a \"type\": \"function\" key and ended with <|eom_id|> (Environment: ipython). The doc declares the tool with an (invalid) list of properties typed 'object'; the schema here keeps the names and descriptions without types."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-multi-turn-second-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-multi-turn-second-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['convert_units'], got []
expected_matchspecial failtool_calls: expected ['convert_units'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['convert_units'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['convert_units'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['convert_units'] | rand:1:8 (fail): tool_calls: one [], got ['convert_units'] | rand:2:8 (fail): tool_calls: one [], got ['convert_units'] | rand:3:8 (fail): tool_calls: one [], got ['convert_units'] | rand:4:8 (fail): tool_calls: one [], got ['convert_units'] | rand:5:8 (fail): tool_calls: one [], got ['convert_units']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-multi-turn-second-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 11 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,13 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "from_unit": "celsius",
-        "to_unit": "fahrenheit",
-        "value": 21.5
-      },
-      "name": "convert_units"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "from_unit": "celsius",
        "to_unit": "fahrenheit",
        "value": 21.5
      },
      "name": "convert_units"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, multi-turn, x-parameters-key, x-llama-json.

Raw output

{"name": "convert_units", "parameters": {"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "from_unit": "celsius",
        "to_unit": "fahrenheit",
        "value": 21.5
      },
      "name": "convert_units"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-multi-turn-second-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"convert_units\", \"parameters\": {\"value\": 21.5, \"from_unit\": \"celsius\", \"to_unit\": \"fahrenheit\"}}", "output_token_ids": [5018, 609, 794, 330, 14500, 29445, 498, 330, 14105, 794, 5324, 970, 794, 220, 1691, 13, 20, 11, 330, 1527, 15176, 794, 330, 66, 41347, 498, 330, 998, 15176, 794, 330, 69, 49010, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "convert_units", "arguments": {"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}}]}, "tags": ["single-call", "multi-turn", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-nested-json expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['update_settings'], got []
expected_matchspecial failtool_calls: expected ['update_settings'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['update_settings'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['update_settings'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['update_settings'] | rand:1:8 (fail): tool_calls: one [], got ['update_settings'] | rand:2:8 (fail): tool_calls: one [], got ['update_settings'] | rand:3:8 (fail): tool_calls: one [], got ['update_settings'] | rand:4:8 (fail): tool_calls: one [], got ['update_settings'] | rand:5:8 (fail): tool_calls: one [], got ['update_settings']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-nested-json --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,25 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "dry_run": false,
-        "settings": {
-          "notifications": {
-            "email": true,
-            "sms": false
-          },
-          "theme": {
-            "accent": [
-              255,
-              128,
-              0
-            ],
-            "mode": "dark"
-          }
-        }
-      },
-      "name": "update_settings"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, nested-json, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "update_settings", "parameters": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-nested-json", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"update_settings\", \"parameters\": {\"settings\": {\"theme\": {\"mode\": \"dark\", \"accent\": [255, 128, 0]}, \"notifications\": {\"email\": true, \"sms\": false}}, \"dry_run\": false}}", "output_token_ids": [5018, 609, 794, 330, 2443, 11090, 498, 330, 14105, 794, 5324, 6648, 794, 5324, 9224, 794, 5324, 8684, 794, 330, 23449, 498, 330, 78648, 794, 510, 3192, 11, 220, 4386, 11, 220, 15, 67225, 330, 39288, 794, 5324, 2386, 794, 837, 11, 330, 42492, 794, 905, 39254, 330, 54310, 14334, 794, 905, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "update_settings", "arguments": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}]}, "tags": ["single-call", "nested-json", "numeric-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-sglang-python-tag-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-sglang-python-tag-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-python-tag-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Beijing"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76. Tags: single-call, x-python-tag-prefix, x-arguments-key, x-llama-json.

Raw output

<|python_tag|>{"name": "get_weather", "arguments": {"city": "Beijing"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-sglang-python-tag-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "<|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Beijing\"}}", "output_token_ids": [128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 3513, 24570, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Beijing"}}]}, "tags": ["single-call", "x-python-tag-prefix", "x-arguments-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-sglang-text-before-python-tag expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-sglang-text-before-python-tag
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Let me check. ', got None
expected_matchone failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchspecial failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchtoken failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchrand:1:8 failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchrand:2:8 failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchrand:3:8 failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchrand:4:8 failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
expected_matchrand:5:8 failcontent: expected 'Let me check. ', got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamtoken failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:1:8 failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:2:8 failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:3:8 failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:4:8 failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:5:8 failcontent: nonstream None, got 'Let me check. {"name": "get_weather", "arguments": {"city": "Tokyo"}}'; tool_calls: nonstream ['get_weather'], got []

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-text-before-python-tag --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check. ",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Let me check. ",
+  "content": "Let me check. {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88. Tags: single-call, text-before-call, x-python-tag-prefix, x-arguments-key, x-text-plus-call, x-llama-json.

Raw output

Let me check. <|python_tag|>{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": "Let me check. ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-sglang-text-before-python-tag", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "Let me check. <|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [10267, 757, 1817, 13, 220, 128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "Let me check. ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-python-tag-prefix", "x-arguments-key", "x-text-plus-call", "x-llama-json"], "notes": "SGLang asserts the normal text keeps its trailing space."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, x-parameters-key, x-llama-json.

Raw output

{"name": "get_weather", "parameters": {"city": "Paris", "unit": "celsius"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-single-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"get_weather\", \"parameters\": {\"city\": \"Paris\", \"unit\": \"celsius\"}}", "output_token_ids": [5018, 609, 794, 330, 456, 70464, 498, 330, 14105, 794, 5324, 9103, 794, 330, 60704, 498, 330, 3928, 794, 330, 66, 41347, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "celsius"}}]}, "tags": ["single-call", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-string-escapes expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-string-escapes
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['write_file'], got []
expected_matchspecial failtool_calls: expected ['write_file'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['write_file'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['write_file'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['write_file'] | rand:1:8 (fail): tool_calls: one [], got ['write_file'] | rand:2:8 (fail): tool_calls: one [], got ['write_file'] | rand:3:8 (fail): tool_calls: one [], got ['write_file'] | rand:4:8 (fail): tool_calls: one [], got ['write_file'] | rand:5:8 (fail): tool_calls: one [], got ['write_file']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-string-escapes --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
-        "path": "C:\\Users\\ana\\notes.txt"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
        "path": "C:\\Users\\ana\\notes.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "C:\\Users\\ana\\notes.txt", "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
        "path": "C:\\Users\\ana\\notes.txt"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-string-escapes", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"C:\\\\Users\\\\ana\\\\notes.txt\", \"content\": \"line 1\\n\\tline 2 \\\"quoted\\\" {braces} \\\\ end\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 34, 24754, 7283, 3505, 3444, 3505, 18790, 3996, 498, 330, 1834, 794, 330, 1074, 220, 16, 1734, 5061, 1074, 220, 17, 7393, 64825, 2153, 314, 1347, 2492, 92, 26033, 842, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "C:\\Users\\ana\\notes.txt", "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end"}}]}, "tags": ["single-call", "string-escapes", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-truncated-inside-parameters expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-truncated-inside-parameters
CheckStrategyResultDetail
expected_errortoken failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:1:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:2:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:3:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:4:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:5:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
stream_equals_nonstreamone failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None
stream_equals_nonstreamspecial failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None
stream_equals_nonstreamtoken failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
split_invariance* failvs one: token (fail): tool_calls: one [], got ['create_event'] | rand:1:8 (fail): tool_calls: one [], got ['create_event'] | rand:2:8 (fail): tool_calls: one [], got ['create_event'] | rand:3:8 (fail): tool_calls: one [], got ['create_event'] | rand:4:8 (fail): tool_calls: one [], got ['create_event'] | rand:5:8 (fail): tool_calls: one [], got ['create_event']
arguments_jsontoken fail[0] create_event: arguments are not valid JSON text ('{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"]': Expecting ',' delimiter: line 1 column 72 (char 71))
arguments_jsonrand:3:8 fail[0] create_event: arguments are not valid JSON text ('{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"]': Expecting ',' delimiter: line 1 column 72 (char 71))
arguments_schematoken fail[0] create_event: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/truncated.jsonl --id llama/l3-truncated-inside-parameters --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/llama/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: token, rand:3:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"]"
      },
      "name": "create_event"
    }
  ]
}

Strategies: rand:1:8, rand:2:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}

Strategies: nonstream

{
  "content": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: truncated, x-llama-json.

Expected graceful failure: max_tokens hit inside the parameters object; the JSON never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-truncated-inside-parameters", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,", "output_token_ids": [5018, 609, 794, 330, 3261, 6891, 498, 330, 14105, 794, 5324, 2150, 794, 330, 49833, 455, 498, 330, 57558, 5633, 794, 4482, 3444, 36587, 916, 498, 330, 754, 36587, 916, 8073, 330, 17456, 68755, 794, 220, 868, 11], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected_error": {"reason": "max_tokens hit inside the parameters object; the JSON never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-llama-json"], "notes": "Token prefix of llama/l3-typed-values."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-typed-values expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-typed-values
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['create_event'], got []
expected_matchspecial failtool_calls: expected ['create_event'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['create_event'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['create_event'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['create_event'] | rand:1:8 (fail): tool_calls: one [], got ['create_event'] | rand:2:8 (fail): tool_calls: one [], got ['create_event'] | rand:3:8 (fail): tool_calls: one [], got ['create_event'] | rand:4:8 (fail): tool_calls: one [], got ['create_event'] | rand:5:8 (fail): tool_calls: one [], got ['create_event']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-typed-values --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,18 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "all_day": false,
-        "attendees": [
-          "ana@example.com",
-          "bo@example.com"
-        ],
-        "duration_minutes": 15,
-        "reminder_minutes": null,
-        "title": "Standup"
-      },
-      "name": "create_event"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "all_day": false,
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "reminder_minutes": null,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "all_day": false,
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "reminder_minutes": null,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-typed-values", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15, \"all_day\": false, \"reminder_minutes\": null}}", "output_token_ids": [5018, 609, 794, 330, 3261, 6891, 498, 330, 14105, 794, 5324, 2150, 794, 330, 49833, 455, 498, 330, 57558, 5633, 794, 4482, 3444, 36587, 916, 498, 330, 754, 36587, 916, 8073, 330, 17456, 68755, 794, 220, 868, 11, 330, 543, 17181, 794, 905, 11, 330, 79005, 68755, 794, 854, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}}]}, "tags": ["single-call", "numeric-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-unicode-emoji expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['search_web'], got []
expected_matchspecial failtool_calls: expected ['search_web'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['search_web'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['search_web'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['search_web'] | rand:1:8 (fail): tool_calls: one [], got ['search_web'] | rand:2:8 (fail): tool_calls: one [], got ['search_web'] | rand:3:8 (fail): tool_calls: one [], got ['search_web'] | rand:4:8 (fail): tool_calls: one [], got ['search_web'] | rand:5:8 (fail): tool_calls: one [], got ['search_web']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-unicode-emoji --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "Zürich café ☕ 東京の天気 🌸"
-      },
-      "name": "search_web"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, unicode, x-parameters-key, x-llama-json.

Raw output

{"name": "search_web", "parameters": {"query": "Zürich café ☕ 東京の天気 🌸"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-unicode-emoji", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"search_web\", \"parameters\": {\"query\": \"Zürich café ☕ 東京の天気 🌸\"}}", "output_token_ids": [5018, 609, 794, 330, 1874, 27050, 498, 330, 14105, 794, 5324, 1663, 794, 330, 57, 5297, 718, 53050, 26182, 243, 119109, 16144, 36827, 95221, 11410, 234, 116, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search_web", "arguments": {"query": "Zürich café ☕ 東京の天気 🌸"}}]}, "tags": ["single-call", "unicode", "x-parameters-key", "x-llama-json"], "notes": "The template uses tojson with ensure_ascii=False, so non-ASCII text is literal."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-arguments-key expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-vllm-arguments-key
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['searchTool'], got []
expected_matchspecial failtool_calls: expected ['searchTool'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['searchTool'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['searchTool'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['searchTool'] | rand:1:8 (fail): tool_calls: one [], got ['searchTool'] | rand:2:8 (fail): tool_calls: one [], got ['searchTool'] | rand:3:8 (fail): tool_calls: one [], got ['searchTool'] | rand:4:8 (fail): tool_calls: one [], got ['searchTool'] | rand:5:8 (fail): tool_calls: one [], got ['searchTool']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-arguments-key --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "test"
-      },
-      "name": "searchTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "test"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84. Tags: single-call, x-arguments-key, x-llama-json.

Raw output

{"name": "searchTool", "arguments": {"query": "test"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "test"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-arguments-key", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"arguments\": {\"query\": \"test\"}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 16774, 794, 5324, 1663, 794, 330, 1985, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test"}}]}, "tags": ["single-call", "x-arguments-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. Llama's documented key is 'parameters'; models also emit 'arguments', which vLLM and SGLang accept."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-brackets-in-strings expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-vllm-brackets-in-strings
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['searchTool'], got []
expected_matchspecial failtool_calls: expected ['searchTool'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['searchTool'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['searchTool'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['searchTool'] | rand:1:8 (fail): tool_calls: one [], got ['searchTool'] | rand:2:8 (fail): tool_calls: one [], got ['searchTool'] | rand:3:8 (fail): tool_calls: one [], got ['searchTool'] | rand:4:8 (fail): tool_calls: one [], got ['searchTool'] | rand:5:8 (fail): tool_calls: one [], got ['searchTool']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-brackets-in-strings --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,14 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "nested": {
-          "inner": "more {brackets}"
-        },
-        "query": "test {value} [complex]"
-      },
-      "name": "searchTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "nested": {
          "inner": "more {brackets}"
        },
        "query": "test {value} [complex]"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L195-L214. Tags: single-call, nested-json, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "searchTool", "parameters": {"query": "test {value} [complex]","nested": {"inner": "more {brackets}"}}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "nested": {
          "inner": "more {brackets}"
        },
        "query": "test {value} [complex]"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-brackets-in-strings", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L195-L214", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "nested": {"type": "object"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"parameters\": {\"query\": \"test {value} [complex]\",\"nested\": {\"inner\": \"more {brackets}\"}}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 14105, 794, 5324, 1663, 794, 330, 1985, 314, 970, 92, 510, 24126, 60, 2247, 60371, 794, 5324, 4481, 794, 330, 6518, 314, 1347, 18889, 10064, 76642], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}}]}, "tags": ["single-call", "nested-json", "string-escapes", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. Braces and brackets inside strings must not confuse brace matching."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-deeply-nested expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-vllm-deeply-nested
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['complexTool'], got []
expected_matchspecial failtool_calls: expected ['complexTool'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['complexTool'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['complexTool'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['complexTool'] | rand:1:8 (fail): tool_calls: one [], got ['complexTool'] | rand:2:8 (fail): tool_calls: one [], got ['complexTool'] | rand:3:8 (fail): tool_calls: one [], got ['complexTool'] | rand:4:8 (fail): tool_calls: one [], got ['complexTool'] | rand:5:8 (fail): tool_calls: one [], got ['complexTool']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-deeply-nested --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,19 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "level1": {
-          "level2": {
-            "level3": {
-              "level4": {
-                "value": "deep"
-              }
-            }
-          }
-        }
-      },
-      "name": "complexTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "level1": {
          "level2": {
            "level3": {
              "level4": {
                "value": "deep"
              }
            }
          }
        }
      },
      "name": "complexTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L146-L167. Tags: single-call, nested-json, x-parameters-key, x-llama-json.

Raw output

{"name": "complexTool", "parameters": {"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "level1": {
          "level2": {
            "level3": {
              "level4": {
                "value": "deep"
              }
            }
          }
        }
      },
      "name": "complexTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-deeply-nested", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L146-L167", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "complexTool", "parameters": {"type": "object", "properties": {"level1": {"type": "object"}}}}}], "raw_output": "{\"name\": \"complexTool\", \"parameters\": {\"level1\": {\"level2\": {\"level3\": {\"level4\": {\"value\": \"deep\"}}}}}}", "output_token_ids": [5018, 609, 794, 330, 24126, 7896, 498, 330, 14105, 794, 5324, 3374, 16, 794, 5324, 3374, 17, 794, 5324, 3374, 18, 794, 5324, 3374, 19, 794, 5324, 970, 794, 330, 33980, 32075, 3500, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "complexTool", "arguments": {"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}}]}, "tags": ["single-call", "nested-json", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-escaped-quotes expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-vllm-escaped-quotes
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['parserTool'], got []
expected_matchspecial failtool_calls: expected ['parserTool'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['parserTool'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['parserTool'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['parserTool'] | rand:1:8 (fail): tool_calls: one [], got ['parserTool'] | rand:2:8 (fail): tool_calls: one [], got ['parserTool'] | rand:3:8 (fail): tool_calls: one [], got ['parserTool'] | rand:4:8 (fail): tool_calls: one [], got ['parserTool'] | rand:5:8 (fail): tool_calls: one [], got ['parserTool']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-escaped-quotes --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "He said \"Hello {world}\""
-      },
-      "name": "parserTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "He said \"Hello {world}\""
      },
      "name": "parserTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L217-L231. Tags: single-call, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "parserTool", "parameters": {"text": "He said \"Hello {world}\""}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "He said \"Hello {world}\""
      },
      "name": "parserTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-escaped-quotes", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L217-L231", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "parserTool", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}}}}], "raw_output": "{\"name\": \"parserTool\", \"parameters\": {\"text\": \"He said \\\"Hello {world}\\\"\"}}", "output_token_ids": [5018, 609, 794, 330, 9854, 7896, 498, 330, 14105, 794, 5324, 1342, 794, 330, 1548, 1071, 7393, 9906, 314, 14957, 47969, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "parserTool", "arguments": {"text": "He said \"Hello {world}\""}}]}, "tags": ["single-call", "string-escapes", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-json-without-name-is-content expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-vllm-json-without-name-is-content
CheckStrategyResultDetail
expected_matchone failcontent: expected '{"parameters": {}}', got None
expected_matchspecial failcontent: expected '{"parameters": {}}', got None
expected_matchtoken failcontent: expected '{"parameters": {}}', got None
expected_matchrand:1:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:2:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:3:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:4:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:5:8 failcontent: expected '{"parameters": {}}', got None
stream_equals_nonstreamone failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamspecial failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamtoken failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"parameters": {}}', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"parameters": {}}', got None

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-json-without-name-is-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"parameters\": {}}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Strategies: nonstream

Matches the expected parse.

{
  "content": "{\"parameters\": {}}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"parameters": {}}

Expected parse

{
  "content": "{\"parameters\": {}}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-json-without-name-is-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"parameters\": {}}", "output_token_ids": [5018, 14105, 794, 314, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"parameters\": {}}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. A JSON object without a 'name' is not a call; the text is returned unchanged as content."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l3-vllm-parameters-key expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for llama/l3-vllm-parameters-key
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['searchTool'], got []
expected_matchspecial failtool_calls: expected ['searchTool'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['searchTool'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['searchTool'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['searchTool'] | rand:1:8 (fail): tool_calls: one [], got ['searchTool'] | rand:2:8 (fail): tool_calls: one [], got ['searchTool'] | rand:3:8 (fail): tool_calls: one [], got ['searchTool'] | rand:4:8 (fail): tool_calls: one [], got ['searchTool'] | rand:5:8 (fail): tool_calls: one [], got ['searchTool']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-parameters-key --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "limit": 10,
-        "query": "test query"
-      },
-      "name": "searchTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "limit": 10,
        "query": "test query"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L42-L53. Tags: single-call, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "searchTool", "parameters": {"query": "test query", "limit": 10}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "limit": 10,
        "query": "test query"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-parameters-key", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L42-L53", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"parameters\": {\"query\": \"test query\", \"limit\": 10}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 14105, 794, 5324, 1663, 794, 330, 1985, 3319, 498, 330, 9696, 794, 220, 605, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test query", "limit": 10}}]}, "tags": ["single-call", "numeric-arguments", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "model_rule": "Llama-3",
  "model_type": "llama",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "se the tools to answer.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama3_json",
  "units_source": "fixture.output_token_ids"
}
fail llama/l4-bug-leading-underscore-identifier expected_match, stream_equals_nonstream
Checks that did not pass for llama/l4-bug-leading-underscore-identifier
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
stream_equals_nonstreamone failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamspecial failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamtoken failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '[search(_limit=5)]', got None; tool_calls: nonstream [], got ['search']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/imported.jsonl --id llama/l4-bug-leading-underscore-identifier --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 18 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "[search(_limit=5)]",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "_limit": 5
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "_limit": 5
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/56840. Tags: single-call, regression, x-leading-underscore-identifier, x-llama-pythonic.

Raw output

[search(_limit=5)]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "_limit": 5
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l4-bug-leading-underscore-identifier", "family": "llama", "models": ["meta-llama/Llama-4-Scout-17B-16E-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/56840", "revision": "issue opened 2026-09-14", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's OUTPUTS list."}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"_limit": {"type": "integer"}}}}}], "raw_output": "[search(_limit=5)]", "output_token_ids": [71, 5546, 5810, 22739, 41, 33, 12432], "tokenizer": {"repo": "unsloth/Llama-4-Scout-17B-16E-Instruct", "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"_limit": 5}}]}, "tags": ["single-call", "regression", "x-leading-underscore-identifier", "x-llama-pythonic"], "notes": "Valid Python; JSON-Schema property names may start with '_'. vLLM's non-streaming pythonic regex rejected it and returned the text as content, while streaming produced the call."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|header_start|>assistant<|header_end|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "model_rule": "Llama-4",
  "model_type": "llama4",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "_end|>\n\nUse the tools to answer.<|eot|><|header_start|>assistant<|header_end|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "93c6a232af67e2b3bc9218179ec3fea5176ec88b2a4a4fb66e51feb2ef63c0bd",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-4-Scout-17B-16E-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama4_pythonic",
  "units_source": "fixture.output_token_ids"
}
fail llama/l4-truncated-missing-close-bracket expected_error, stream_equals_nonstream, arguments_json, arguments_schema
Checks that did not pass for llama/l4-truncated-missing-close-bracket
CheckStrategyResultDetail
expected_errorone failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorspecial failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errortoken failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorrand:1:8 failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorrand:2:8 failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorrand:3:8 failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorrand:4:8 failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
expected_errorrand:5:8 failreturned 2 tool call(s) ['get_weather', 'get_weather'] for The pythonic call list is cut before its final string, call and list close.
stream_equals_nonstreamone failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamspecial failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamtoken failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None; tool_calls: nonstream [], got ['get_weather', 'get_weather']
arguments_jsonone fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonspecial fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsontoken fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[1] get_weather: arguments are not valid JSON text ('{"city": "Seattle': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemaone fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[1] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[1] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/llama/recorded.jsonl --id llama/l4-truncated-missing-close-bracket --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/llama/recorded.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "San Francisco"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Seattle"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "[get_weather(city=\"San Francisco\"), get_weather(city=\"Seattle",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: spec_example, https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama4/prompt_format.md#L204-L207. Tags: truncated, malformed, x-llama-pythonic, x-derived-truncation.

Expected graceful failure: The pythonic call list is cut before its final string, call and list close. (accept: no_tool_calls, content_passthrough, exception).

Raw output

[get_weather(city="San Francisco"), get_weather(city="Seattle
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l4-truncated-missing-close-bracket", "family": "llama", "models": ["meta-llama/Llama-4-Scout-17B-16E-Instruct"], "spec_version": "0.1", "provenance": {"kind": "spec_example", "source_url": "https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama4/prompt_format.md#L204-L207", "revision": "0e0b8c519242d5833d8c11bffc1232b77ad7f301", "license": "LicenseRef-llama4-community", "generator": "scripts/fixtures/llama/build.py", "attribution": "Llama 4 is licensed under the Llama 4 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. Short example quoted with attribution (THIRD_PARTY_NOTICES.md)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather info for places", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The name of the city to get the weather for"}, "metric": {"type": "string", "description": "The metric for weather. Options are: celsius, fahrenheit", "default": "celsius"}}, "required": ["city"]}}}], "raw_output": "[get_weather(city=\"San Francisco\"), get_weather(city=\"Seattle", "output_token_ids": [193913, 191381, 103469, 508, 33300, 20748, 9811, 872, 191381, 103469, 508, 190860], "tokenizer": {"repo": "unsloth/Llama-4-Scout-17B-16E-Instruct", "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b", "mode": "hf"}, "expected_error": {"reason": "The pythonic call list is cut before its final string, call and list close.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "malformed", "x-llama-pythonic", "x-derived-truncation"], "notes": "Derived, not quoted: the token prefix of llama/l4-meta-pythonic-parallel (a spec_example) without its last token (which carries the closing quote, parenthesis and bracket): the 'missing bracket' failure mode reported in https://github.com/vllm-project/vllm/issues/30722."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|header_start|>assistant<|header_end|>\n\n",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "model_rule": "Llama-4",
  "model_type": "llama4",
  "parser_source": "docs/formats/llama.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "_end|>\n\nUse the tools to answer.<|eot|><|header_start|>assistant<|header_end|>\n\n",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "93c6a232af67e2b3bc9218179ec3fea5176ec88b2a4a4fb66e51feb2ef63c0bd",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "unsloth/Llama-4-Scout-17B-16E-Instruct",
    "requested_mode": "auto",
    "resolved_revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "llama4_pythonic",
  "units_source": "fixture.output_token_ids"
}