CanIToolCall caniuse.com for tool calling

Llama (3.1, 3.2, 3.3, 4) on sglang 0.5.20

fail 26% strict pass 9 pass · 25 fail

Run

Engine
sglang 0.5.20
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}
Results file
data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 9 0 23 0 28%
expected_error 1 0 1 0 50%
stream_equals_nonstream 13 0 21 0 38%
split_invariance 15 0 19 0 44%
no_leakage 33 0 1 0 97%
arguments_json 7 0 19 0 27%
arguments_schema 7 0 19 0 27%
parallel_order 2 0 0 0 100%

Fixtures needing attention

fail llama/l3-bug-empty-json-object-in-content expected_match
Checks that did not pass for llama/l3-bug-empty-json-object-in-content
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected '{}', got None
expected_matchone failcontent: expected '{}', got None
expected_matchspecial failcontent: expected '{}', got None
expected_matchtoken failcontent: expected '{}', got None
expected_matchrand:1:8 failcontent: expected '{}', got None
expected_matchrand:2:8 failcontent: expected '{}', got None
expected_matchrand:3:8 failcontent: expected '{}', got None
expected_matchrand:4:8 failcontent: expected '{}', got None
expected_matchrand:5:8 failcontent: expected '{}', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-empty-json-object-in-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{}

Expected parse

{
  "content": "{}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-empty-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue: input '{}' -> content '' (the message is gone)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{}", "output_token_ids": [6390], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-bug-leading-json-object-in-content expected_match, stream_equals_nonstream
Checks that did not pass for llama/l3-bug-leading-json-object-in-content
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected '{"a": 1} is a dict', got 'is a dict'
expected_matchone failcontent: expected '{"a": 1} is a dict', got None
expected_matchspecial failcontent: expected '{"a": 1} is a dict', got None
expected_matchtoken failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:1:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:2:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:3:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:4:8 failcontent: expected '{"a": 1} is a dict', got None
expected_matchrand:5:8 failcontent: expected '{"a": 1} is a dict', got None
stream_equals_nonstreamone failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamspecial failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamtoken failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'is a dict', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'is a dict', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-leading-json-object-in-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 15 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"a\": 1} is a dict",
+  "content": "is a dict",
   "reasoning_content": null,
   "tool_calls": []

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"a\": 1} is a dict",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"a": 1} is a dict

Expected parse

{
  "content": "{\"a\": 1} is a dict",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-leading-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's reproduction (Llama-3.2-1B-Instruct, tools=[get_weather])."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{\"a\": 1} is a dict", "output_token_ids": [5018, 64, 794, 220, 16, 92, 374, 264, 6587], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"a\": 1} is a dict", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "Not a call (no name/parameters). SGLang's llama3 parser deleted the leading JSON and returned 'is a dict'. The issue's expectation: when no call is found the text must be unchanged."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-bug-whole-call-single-delta expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-bug-whole-call-single-delta
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Tokyo"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Tokyo"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Tokyo"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Tokyo"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Tokyo"}'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-whole-call-single-delta --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "city": "Tokyo"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/48294. Tags: single-call, regression, x-single-delta, x-arguments-key, x-llama-json.

Raw output

{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-bug-whole-call-single-delta", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/48294", "revision": "issue opened 2026-07-11", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's minimal reproduction for llama3_json."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "regression", "x-single-delta", "x-arguments-key", "x-llama-json"], "notes": "vLLM's llama3_json streaming path emitted nothing when this whole message arrived in one delta (the 'one' chunking strategy replays exactly that)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail llama/l3-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-empty-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{}'
arguments_jsonone fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-empty-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, empty-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "get_time", "parameters": {}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-empty-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"get_time\", \"parameters\": {}}", "output_token_ids": [5018, 609, 794, 330, 456, 3084, 498, 330, 14105, 794, 314, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-json-answer-not-a-call expected_match
Checks that did not pass for llama/l3-json-answer-not-a-call
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchone failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchspecial failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchtoken failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:1:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:2:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:3:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:4:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None
expected_matchrand:5:8 failcontent: expected '{"capital": "Paris", "country": "France"}', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-json-answer-not-a-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"capital": "Paris", "country": "France"}

Expected parse

{
  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-json-answer-not-a-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"capital\": \"Paris\", \"country\": \"France\"}", "output_token_ids": [5018, 66163, 794, 330, 60704, 498, 330, 11389, 794, 330, 50100, 9388], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"capital\": \"Paris\", \"country\": \"France\"}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "A plain JSON answer: it has no name/parameters keys, so it is content, not a call (cf. https://github.com/sgl-project/sglang/issues/35562)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-long-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-long-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown..., got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown..., got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown..., got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown..., got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown... | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown... | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown... | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown... | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown... | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown...
arguments_jsonone fail[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-long-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
-        "path": "fox.txt"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "write_file"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
        "path": "fox.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, long-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog."}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog.",
        "path": "fox.txt"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-long-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"fox.txt\", \"content\": \"000: The quick brown fox jumps over the lazy dog.\\n001: The quick brown fox jumps over the lazy dog.\\n002: The quick brown fox jumps over the lazy dog.\\n003: The quick brown fox jumps over the lazy dog.\\n004: The quick brown fox jumps over the lazy dog.\\n005: The quick brown fox jumps over the lazy dog.\\n006: The quick brown fox jumps over the lazy dog.\\n007: The quick brown fox jumps over the lazy dog.\\n008: The quick brown fox jumps over the lazy dog.\\n009: The quick brown fox jumps over the lazy dog.\\n010: The quick brown fox jumps over the lazy dog.\\n011: The quick brown fox jumps over the lazy dog.\\n012: The quick brown fox jumps over the lazy dog.\\n013: The quick brown fox jumps over the lazy dog.\\n014: The quick brown fox jumps over the lazy dog.\\n015: The quick brown fox jumps over the lazy dog.\\n016: The quick brown fox jumps over the lazy dog.\\n017: The quick brown fox jumps over the lazy dog.\\n018: The quick brown fox jumps over the lazy dog.\\n019: The quick brown fox jumps over the lazy dog.\\n020: The quick brown fox jumps over the lazy dog.\\n021: The quick brown fox jumps over the lazy dog.\\n022: The quick brown fox jumps over the lazy dog.\\n023: The quick brown fox jumps over the lazy dog.\\n024: The quick brown fox jumps over the lazy dog.\\n025: The quick brown fox jumps over the lazy dog.\\n026: The quick brown fox jumps over the lazy dog.\\n027: The quick brown fox jumps over the lazy dog.\\n028: The quick brown fox jumps over the lazy dog.\\n029: The quick brown fox jumps over the lazy dog.\\n030: The quick brown fox jumps over the lazy dog.\\n031: The quick brown fox jumps over the lazy dog.\\n032: The quick brown fox jumps over the lazy dog.\\n033: The quick brown fox jumps over the lazy dog.\\n034: The quick brown fox jumps over the lazy dog.\\n035: The quick brown fox jumps over the lazy dog.\\n036: The quick brown fox jumps over the lazy dog.\\n037: The quick brown fox jumps over the lazy dog.\\n038: The quick brown fox jumps over the lazy dog.\\n039: The quick brown fox jumps over the lazy dog.\\n040: The quick brown fox jumps over the lazy dog.\\n041: The quick brown fox jumps over the lazy dog.\\n042: The quick brown fox jumps over the lazy dog.\\n043: The quick brown fox jumps over the lazy dog.\\n044: The quick brown fox jumps over the lazy dog.\\n045: The quick brown fox jumps over the lazy dog.\\n046: The quick brown fox jumps over the lazy dog.\\n047: The quick brown fox jumps over the lazy dog.\\n048: The quick brown fox jumps over the lazy dog.\\n049: The quick brown fox jumps over the lazy dog.\\n050: The quick brown fox jumps over the lazy dog.\\n051: The quick brown fox jumps over the lazy dog.\\n052: The quick brown fox jumps over the lazy dog.\\n053: The quick brown fox jumps over the lazy dog.\\n054: The quick brown fox jumps over the lazy dog.\\n055: The quick brown fox jumps over the lazy dog.\\n056: The quick brown fox jumps over the lazy dog.\\n057: The quick brown fox jumps over the lazy dog.\\n058: The quick brown fox jumps over the lazy dog.\\n059: The quick brown fox jumps over the lazy dog.\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 15361, 3996, 498, 330, 1834, 794, 330, 931, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 4119, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 6726, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 6268, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 8759, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 8504, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11030, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11194, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11436, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 13858, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 7755, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 10731, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11531, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16368, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 15901, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16037, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 15794, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 17248, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16745, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18089, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11139, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 11592, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18642, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20063, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 19592, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18070, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21641, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21360, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22000, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23273, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 14649, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 18887, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21040, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 13103, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22379, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22407, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23110, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23587, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24462, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 21602, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 12505, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20945, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22349, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 17776, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 20078, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23785, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24222, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24970, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23904, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25307, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 16193, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 23545, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24130, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25210, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25230, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 22913, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 25921, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 26866, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 24824, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 7255, 77, 27033, 25, 578, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 1210, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "fox.txt", "content": "000: The quick brown fox jumps over the lazy dog.\n001: The quick brown fox jumps over the lazy dog.\n002: The quick brown fox jumps over the lazy dog.\n003: The quick brown fox jumps over the lazy dog.\n004: The quick brown fox jumps over the lazy dog.\n005: The quick brown fox jumps over the lazy dog.\n006: The quick brown fox jumps over the lazy dog.\n007: The quick brown fox jumps over the lazy dog.\n008: The quick brown fox jumps over the lazy dog.\n009: The quick brown fox jumps over the lazy dog.\n010: The quick brown fox jumps over the lazy dog.\n011: The quick brown fox jumps over the lazy dog.\n012: The quick brown fox jumps over the lazy dog.\n013: The quick brown fox jumps over the lazy dog.\n014: The quick brown fox jumps over the lazy dog.\n015: The quick brown fox jumps over the lazy dog.\n016: The quick brown fox jumps over the lazy dog.\n017: The quick brown fox jumps over the lazy dog.\n018: The quick brown fox jumps over the lazy dog.\n019: The quick brown fox jumps over the lazy dog.\n020: The quick brown fox jumps over the lazy dog.\n021: The quick brown fox jumps over the lazy dog.\n022: The quick brown fox jumps over the lazy dog.\n023: The quick brown fox jumps over the lazy dog.\n024: The quick brown fox jumps over the lazy dog.\n025: The quick brown fox jumps over the lazy dog.\n026: The quick brown fox jumps over the lazy dog.\n027: The quick brown fox jumps over the lazy dog.\n028: The quick brown fox jumps over the lazy dog.\n029: The quick brown fox jumps over the lazy dog.\n030: The quick brown fox jumps over the lazy dog.\n031: The quick brown fox jumps over the lazy dog.\n032: The quick brown fox jumps over the lazy dog.\n033: The quick brown fox jumps over the lazy dog.\n034: The quick brown fox jumps over the lazy dog.\n035: The quick brown fox jumps over the lazy dog.\n036: The quick brown fox jumps over the lazy dog.\n037: The quick brown fox jumps over the lazy dog.\n038: The quick brown fox jumps over the lazy dog.\n039: The quick brown fox jumps over the lazy dog.\n040: The quick brown fox jumps over the lazy dog.\n041: The quick brown fox jumps over the lazy dog.\n042: The quick brown fox jumps over the lazy dog.\n043: The quick brown fox jumps over the lazy dog.\n044: The quick brown fox jumps over the lazy dog.\n045: The quick brown fox jumps over the lazy dog.\n046: The quick brown fox jumps over the lazy dog.\n047: The quick brown fox jumps over the lazy dog.\n048: The quick brown fox jumps over the lazy dog.\n049: The quick brown fox jumps over the lazy dog.\n050: The quick brown fox jumps over the lazy dog.\n051: The quick brown fox jumps over the lazy dog.\n052: The quick brown fox jumps over the lazy dog.\n053: The quick brown fox jumps over the lazy dog.\n054: The quick brown fox jumps over the lazy dog.\n055: The quick brown fox jumps over the lazy dog.\n056: The quick brown fox jumps over the lazy dog.\n057: The quick brown fox jumps over the lazy dog.\n058: The quick brown fox jumps over the lazy dog.\n059: The quick brown fox jumps over the lazy dog."}}]}, "tags": ["single-call", "long-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema
Checks that did not pass for llama/l3-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea...; tool_calls: expected ['write_file'], got []
expected_matchone failtool_calls: expected ['write_file'], got []
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got ''
expected_matchtoken failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with"}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start"}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with"}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": ""}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with"}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with"}'
stream_equals_nonstreamone failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None
stream_equals_nonstreamspecial failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamtoken failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_hea..., got None; tool_calls: nonstream [], got ['write_file']
split_invariance* failvs one: special (fail): tool_calls: one [], got ['write_file'] | token (fail): tool_calls: one [], got ['write_file'] | rand:1:8 (fail): tool_calls: one [], got ['write_file'] | rand:2:8 (fail): tool_calls: one [], got ['write_file'] | rand:3:8 (fail): tool_calls: one [], got ['write_file'] | rand:4:8 (fail): tool_calls: one [], got ['write_file'] | rand:5:8 (fail): tool_calls: one [], got ['write_file']
no_leakagenonstream failcontent contains '<|start_header_id|>'; content contains '<|end_header_id|>'
arguments_jsonspecial fail[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaspecial fail[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"docs/llama.md\", \"content\": \"Tool calls may start with ; headers look like <|start_header_id|>ipython<|end_header_id|>.\"}}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
-        "path": "docs/llama.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
-        "path": "docs/llama.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
-        "path": "docs/llama.md"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "write_file"

Strategies: token, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
+        "content": "Tool calls may start with",
         "path": "docs/llama.md"
       },

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
+        "content": "Tool calls may start",
         "path": "docs/llama.md"
       },

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
+        "content": "",
         "path": "docs/llama.md"
       },

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, marker-in-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
        "path": "docs/llama.md"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-marker-in-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"docs/llama.md\", \"content\": \"Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 14452, 14, 657, 3105, 22030, 498, 330, 1834, 794, 330, 7896, 6880, 1253, 1212, 449, 220, 128010, 26, 7247, 1427, 1093, 220, 128006, 23799, 4690, 128007, 1210, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}]}, "tags": ["single-call", "marker-in-arguments", "x-parameters-key", "x-llama-json"], "notes": "The JSON string contains <|python_tag|> and header tokens, tokenized as the special tokens. A parser that splits on <|python_tag|> must not cut inside the string."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-meta-python-tag-type-function expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-meta-python-tag-type-function
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"n": "10", "genre": "all"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"n": "10", "genre": "all"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"n": "10", "genre": "all"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"n": "10", "genre": "all"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"n": "10", "genre": "all"}'
arguments_jsonone fail[0] trending_songs: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] trending_songs: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] trending_songs: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] trending_songs: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/recorded.jsonl --id llama/l3-meta-python-tag-type-function --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/llama/recorded.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "genre": "all",
-        "n": "10"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "trending_songs"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "genre": "all",
        "n": "10"
      },
      "name": "trending_songs"
    }
  ]
}

Fixture

Provenance: spec_example, https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama3_3/prompt_format.md#L403-L412. Tags: single-call, x-llama-json, x-python-tag-prefix, x-extra-type-key, x-eom-terminator.

Raw output

<|python_tag|>{
    "type": "function",
    "name": "trending_songs",
    "parameters": {
        "n": "10",
        "genre": "all"
    }
}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "genre": "all",
        "n": "10"
      },
      "name": "trending_songs"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-meta-python-tag-type-function", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "spec_example", "source_url": "https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama3_3/prompt_format.md#L403-L412", "revision": "0e0b8c519242d5833d8c11bffc1232b77ad7f301", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "attribution": "Llama 3.3 is licensed under the Llama 3.3 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. Short example quoted with attribution (THIRD_PARTY_NOTICES.md)."}, "tools": [{"type": "function", "function": {"name": "trending_songs", "description": "Returns the trending songs on a Music site", "parameters": {"type": "object", "properties": {"n": {"description": "The number of songs to return"}, "genre": {"description": "The genre of the songs to return"}}, "required": ["n"]}}}], "raw_output": "<|python_tag|>{\n    \"type\": \"function\",\n    \"name\": \"trending_songs\",\n    \"parameters\": {\n        \"n\": \"10\",\n        \"genre\": \"all\"\n    }\n}", "output_token_ids": [128010, 517, 262, 330, 1337, 794, 330, 1723, 761, 262, 330, 609, 794, 330, 83, 63094, 646, 13483, 761, 262, 330, 14105, 794, 341, 286, 330, 77, 794, 330, 605, 761, 286, 330, 34713, 794, 330, 543, 702, 262, 457, 92], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "trending_songs", "arguments": {"n": "10", "genre": "all"}}]}, "tags": ["single-call", "x-llama-json", "x-python-tag-prefix", "x-extra-type-key", "x-eom-terminator"], "notes": "Quoted verbatim from the 'Model Response Format' block of Meta's prompt-format doc (spec_example: official format documentation, not a generation captured by this project). The stop token that ends the block is cut, as spec/README.md requires. The model prefixed <|python_tag|>, added a \"type\": \"function\" key and ended with <|eom_id|> (Environment: ipython). The doc declares the tool with an (invalid) list of properties typed 'object'; the schema here keeps the names and descriptions without types."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-multi-turn-second-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-multi-turn-second-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}'
arguments_jsonone fail[0] convert_units: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] convert_units: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] convert_units: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] convert_units: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-multi-turn-second-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,7 +5,5 @@
     {
       "arguments": {
-        "from_unit": "celsius",
-        "to_unit": "fahrenheit",
-        "value": 21.5
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "convert_units"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "from_unit": "celsius",
        "to_unit": "fahrenheit",
        "value": 21.5
      },
      "name": "convert_units"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, multi-turn, x-parameters-key, x-llama-json.

Raw output

{"name": "convert_units", "parameters": {"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "from_unit": "celsius",
        "to_unit": "fahrenheit",
        "value": 21.5
      },
      "name": "convert_units"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-multi-turn-second-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"convert_units\", \"parameters\": {\"value\": 21.5, \"from_unit\": \"celsius\", \"to_unit\": \"fahrenheit\"}}", "output_token_ids": [5018, 609, 794, 330, 14500, 29445, 498, 330, 14105, 794, 5324, 970, 794, 220, 1691, 13, 20, 11, 330, 1527, 15176, 794, 330, 66, 41347, 498, 330, 998, 15176, 794, 330, 69, 49010, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "convert_units", "arguments": {"value": 21.5, "from_unit": "celsius", "to_unit": "fahrenheit"}}]}, "tags": ["single-call", "multi-turn", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-nested-json expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}'
arguments_jsonone fail[0] update_settings: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] update_settings: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] update_settings: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] update_settings: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-nested-json --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,19 +5,5 @@
     {
       "arguments": {
-        "dry_run": false,
-        "settings": {
-          "notifications": {
-            "email": true,
-            "sms": false
-          },
-          "theme": {
-            "accent": [
-              255,
-              128,
-              0
-            ],
-            "mode": "dark"
-          }
-        }
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "update_settings"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, nested-json, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "update_settings", "parameters": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "dry_run": false,
        "settings": {
          "notifications": {
            "email": true,
            "sms": false
          },
          "theme": {
            "accent": [
              255,
              128,
              0
            ],
            "mode": "dark"
          }
        }
      },
      "name": "update_settings"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-nested-json", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"update_settings\", \"parameters\": {\"settings\": {\"theme\": {\"mode\": \"dark\", \"accent\": [255, 128, 0]}, \"notifications\": {\"email\": true, \"sms\": false}}, \"dry_run\": false}}", "output_token_ids": [5018, 609, 794, 330, 2443, 11090, 498, 330, 14105, 794, 5324, 6648, 794, 5324, 9224, 794, 5324, 8684, 794, 330, 23449, 498, 330, 78648, 794, 510, 3192, 11, 220, 4386, 11, 220, 15, 67225, 330, 39288, 794, 5324, 2386, 794, 837, 11, 330, 42492, 794, 905, 39254, 330, 54310, 14334, 794, 905, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "update_settings", "arguments": {"settings": {"theme": {"mode": "dark", "accent": [255, 128, 0]}, "notifications": {"email": true, "sms": false}}, "dry_run": false}}]}, "tags": ["single-call", "nested-json", "numeric-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-sglang-python-tag-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-sglang-python-tag-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Beijing"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Beijing"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Beijing"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Beijing"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Beijing"}'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-python-tag-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "city": "Beijing"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76. Tags: single-call, x-python-tag-prefix, x-arguments-key, x-llama-json.

Raw output

<|python_tag|>{"name": "get_weather", "arguments": {"city": "Beijing"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-sglang-python-tag-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "<|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Beijing\"}}", "output_token_ids": [128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 3513, 24570, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Beijing"}}]}, "tags": ["single-call", "x-python-tag-prefix", "x-arguments-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail llama/l3-sglang-text-before-python-tag expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-sglang-text-before-python-tag
CheckStrategyResultDetail
expected_matchone failcontent: expected 'Let me check. ', got None; tool_calls[0].arguments: expected '{"city": "Tokyo"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Tokyo"}', got ''
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me check. ', got 'Let me check.'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check. ', got 'Let me check.'
stream_equals_nonstreamone failcontent: nonstream 'Let me check. ', got None; tool_calls[0].arguments: nonstream '{"city": "Tokyo"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Tokyo"}', got ''
stream_equals_nonstreamrand:3:8 soft passcontent (whitespace only): nonstream 'Let me check. ', got 'Let me check.'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Let me check. ', got 'Let me check.'
split_invariance* failvs one: special (fail): content: one None, got 'Let me check. ' | token (fail): content: one None, got 'Let me check. '; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:1:8 (fail): content: one None, got 'Let me check. '; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:2:8 (fail): content: one None, got 'Let me check. '; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:3:8 (fail): content: one None, got 'Let me check.'; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:4:8 (fail): content: one None, got 'Let me check.'; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}' | rand:5:8 (fail): content: one None, got 'Let me check. '; tool_calls[0].arguments: one '', got '{"city": "Tokyo"}'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-text-before-python-tag --strategy one --strategy rand:3:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check. ",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Tokyo"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "city": "Tokyo"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check. ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: rand:3:8, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check. ",
+  "content": "Let me check.",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88. Tags: single-call, text-before-call, x-python-tag-prefix, x-arguments-key, x-text-plus-call, x-llama-json.

Raw output

Let me check. <|python_tag|>{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": "Let me check. ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-sglang-text-before-python-tag", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "Let me check. <|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [10267, 757, 1817, 13, 220, 128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "Let me check. ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-python-tag-prefix", "x-arguments-key", "x-text-plus-call", "x-llama-json"], "notes": "SGLang asserts the normal text keeps its trailing space."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail llama/l3-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "celsius"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "celsius"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "celsius"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "celsius"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Paris", "unit": "celsius"}'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Paris",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, x-parameters-key, x-llama-json.

Raw output

{"name": "get_weather", "parameters": {"city": "Paris", "unit": "celsius"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-single-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"get_weather\", \"parameters\": {\"city\": \"Paris\", \"unit\": \"celsius\"}}", "output_token_ids": [5018, 609, 794, 330, 456, 70464, 498, 330, 14105, 794, 5324, 9103, 794, 330, 60704, 498, 330, 3928, 794, 330, 66, 41347, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "celsius"}}]}, "tags": ["single-call", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-string-escapes expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-string-escapes
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"path": "C:\\\\Users\\\\ana\\\\notes.txt", "content": "line 1\\n\\tline 2 \\"quoted\\" {braces} \\\\ end"}'
arguments_jsonone fail[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] write_file: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] write_file: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] write_file: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-string-escapes --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
-        "path": "C:\\Users\\ana\\notes.txt"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "write_file"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
        "path": "C:\\Users\\ana\\notes.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "C:\\Users\\ana\\notes.txt", "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end",
        "path": "C:\\Users\\ana\\notes.txt"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-string-escapes", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"C:\\\\Users\\\\ana\\\\notes.txt\", \"content\": \"line 1\\n\\tline 2 \\\"quoted\\\" {braces} \\\\ end\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 34, 24754, 7283, 3505, 3444, 3505, 18790, 3996, 498, 330, 1834, 794, 330, 1074, 220, 16, 1734, 5061, 1074, 220, 17, 7393, 64825, 2153, 314, 1347, 2492, 92, 26033, 842, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "C:\\Users\\ana\\notes.txt", "content": "line 1\n\tline 2 \"quoted\" {braces} \\ end"}}]}, "tags": ["single-call", "string-escapes", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-truncated-inside-parameters expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-truncated-inside-parameters
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorspecial failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errortoken failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:1:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:2:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:3:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:4:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
expected_errorrand:5:8 failreturned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
stream_equals_nonstreamone failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamspecial failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamtoken failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,', got None; tool_calls: nonstream [], got ['create_event']
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15}'
arguments_jsonone fail[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] create_event: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/truncated.jsonl --id llama/l3-truncated-inside-parameters --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/llama/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "create_event"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}

Strategies: nonstream

{
  "content": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: truncated, x-llama-json.

Expected graceful failure: max_tokens hit inside the parameters object; the JSON never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-truncated-inside-parameters", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,", "output_token_ids": [5018, 609, 794, 330, 3261, 6891, 498, 330, 14105, 794, 5324, 2150, 794, 330, 49833, 455, 498, 330, 57558, 5633, 794, 4482, 3444, 36587, 916, 498, 330, 754, 36587, 916, 8073, 330, 17456, 68755, 794, 220, 868, 11], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected_error": {"reason": "max_tokens hit inside the parameters object; the JSON never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-llama-json"], "notes": "Token prefix of llama/l3-typed-values."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-typed-values expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-typed-values
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}'
arguments_jsonone fail[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] create_event: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] create_event: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-typed-values --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,12 +5,5 @@
     {
       "arguments": {
-        "all_day": false,
-        "attendees": [
-          "ana@example.com",
-          "bo@example.com"
-        ],
-        "duration_minutes": 15,
-        "reminder_minutes": null,
-        "title": "Standup"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "create_event"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "all_day": false,
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "reminder_minutes": null,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "all_day": false,
        "attendees": [
          "ana@example.com",
          "bo@example.com"
        ],
        "duration_minutes": 15,
        "reminder_minutes": null,
        "title": "Standup"
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-typed-values", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15, \"all_day\": false, \"reminder_minutes\": null}}", "output_token_ids": [5018, 609, 794, 330, 3261, 6891, 498, 330, 14105, 794, 5324, 2150, 794, 330, 49833, 455, 498, 330, 57558, 5633, 794, 4482, 3444, 36587, 916, 498, 330, 754, 36587, 916, 8073, 330, 17456, 68755, 794, 220, 868, 11, 330, 543, 17181, 794, 905, 11, 330, 79005, 68755, 794, 854, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15, "all_day": false, "reminder_minutes": null}}]}, "tags": ["single-call", "numeric-arguments", "x-parameters-key", "x-llama-json"]}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-unicode-emoji expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"query": "Zürich café ☕ 東京の天気 🌸"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "Zürich café ☕ 東京の天気 🌸"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"query": "Zürich café ☕ 東京の天気 🌸"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"query": "Zürich café ☕ 東京の天気 🌸"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "Zürich café ☕ 東京の天気 🌸"}'
arguments_jsonone fail[0] search_web: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] search_web: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] search_web: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] search_web: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-unicode-emoji --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "query": "Zürich café ☕ 東京の天気 🌸"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "search_web"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, unicode, x-parameters-key, x-llama-json.

Raw output

{"name": "search_web", "parameters": {"query": "Zürich café ☕ 東京の天気 🌸"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "Zürich café ☕ 東京の天気 🌸"
      },
      "name": "search_web"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-unicode-emoji", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"search_web\", \"parameters\": {\"query\": \"Zürich café ☕ 東京の天気 🌸\"}}", "output_token_ids": [5018, 609, 794, 330, 1874, 27050, 498, 330, 14105, 794, 5324, 1663, 794, 330, 57, 5297, 718, 53050, 26182, 243, 119109, 16144, 36827, 95221, 11410, 234, 116, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search_web", "arguments": {"query": "Zürich café ☕ 東京の天気 🌸"}}]}, "tags": ["single-call", "unicode", "x-parameters-key", "x-llama-json"], "notes": "The template uses tojson with ensure_ascii=False, so non-ASCII text is literal."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 7,
  "version": "0.5.20"
}
fail llama/l3-vllm-arguments-key expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-vllm-arguments-key
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"query": "test"}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "test"}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"query": "test"}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"query": "test"}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"query": "test"}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test"}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test"}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test"}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test"}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test"}'
arguments_jsonone fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] searchTool: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] searchTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-arguments-key --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "query": "test"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "searchTool"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "test"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84. Tags: single-call, x-arguments-key, x-llama-json.

Raw output

{"name": "searchTool", "arguments": {"query": "test"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "test"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-arguments-key", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"arguments\": {\"query\": \"test\"}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 16774, 794, 5324, 1663, 794, 330, 1985, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test"}}]}, "tags": ["single-call", "x-arguments-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. Llama's documented key is 'parameters'; models also emit 'arguments', which vLLM and SGLang accept."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-vllm-brackets-in-strings expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-vllm-brackets-in-strings
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}'
arguments_jsonone fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] searchTool: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] searchTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-brackets-in-strings --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,8 +5,5 @@
     {
       "arguments": {
-        "nested": {
-          "inner": "more {brackets}"
-        },
-        "query": "test {value} [complex]"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "searchTool"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "nested": {
          "inner": "more {brackets}"
        },
        "query": "test {value} [complex]"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L195-L214. Tags: single-call, nested-json, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "searchTool", "parameters": {"query": "test {value} [complex]","nested": {"inner": "more {brackets}"}}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "nested": {
          "inner": "more {brackets}"
        },
        "query": "test {value} [complex]"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-brackets-in-strings", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L195-L214", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "nested": {"type": "object"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"parameters\": {\"query\": \"test {value} [complex]\",\"nested\": {\"inner\": \"more {brackets}\"}}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 14105, 794, 5324, 1663, 794, 330, 1985, 314, 970, 92, 510, 24126, 60, 2247, 60371, 794, 5324, 4481, 794, 330, 6518, 314, 1347, 18889, 10064, 76642], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test {value} [complex]", "nested": {"inner": "more {brackets}"}}}]}, "tags": ["single-call", "nested-json", "string-escapes", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. Braces and brackets inside strings must not confuse brace matching."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-vllm-deeply-nested expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-vllm-deeply-nested
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}'
arguments_jsonone fail[0] complexTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] complexTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] complexTool: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] complexTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-deeply-nested --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,13 +5,5 @@
     {
       "arguments": {
-        "level1": {
-          "level2": {
-            "level3": {
-              "level4": {
-                "value": "deep"
-              }
-            }
-          }
-        }
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "complexTool"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "level1": {
          "level2": {
            "level3": {
              "level4": {
                "value": "deep"
              }
            }
          }
        }
      },
      "name": "complexTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L146-L167. Tags: single-call, nested-json, x-parameters-key, x-llama-json.

Raw output

{"name": "complexTool", "parameters": {"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "level1": {
          "level2": {
            "level3": {
              "level4": {
                "value": "deep"
              }
            }
          }
        }
      },
      "name": "complexTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-deeply-nested", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L146-L167", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "complexTool", "parameters": {"type": "object", "properties": {"level1": {"type": "object"}}}}}], "raw_output": "{\"name\": \"complexTool\", \"parameters\": {\"level1\": {\"level2\": {\"level3\": {\"level4\": {\"value\": \"deep\"}}}}}}", "output_token_ids": [5018, 609, 794, 330, 24126, 7896, 498, 330, 14105, 794, 5324, 3374, 16, 794, 5324, 3374, 17, 794, 5324, 3374, 18, 794, 5324, 3374, 19, 794, 5324, 970, 794, 330, 33980, 32075, 3500, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "complexTool", "arguments": {"level1": {"level2": {"level3": {"level4": {"value": "deep"}}}}}}]}, "tags": ["single-call", "nested-json", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-vllm-escaped-quotes expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-vllm-escaped-quotes
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"text": "He said \\"Hello {world}\\""}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"text": "He said \\"Hello {world}\\""}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"text": "He said \\"Hello {world}\\""}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"text": "He said \\"Hello {world}\\""}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"text": "He said \\"Hello {world}\\""}'
arguments_jsonone fail[0] parserTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] parserTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] parserTool: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] parserTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-escaped-quotes --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "He said \"Hello {world}\""
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "parserTool"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "He said \"Hello {world}\""
      },
      "name": "parserTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L217-L231. Tags: single-call, string-escapes, x-parameters-key, x-llama-json.

Raw output

{"name": "parserTool", "parameters": {"text": "He said \"Hello {world}\""}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "He said \"Hello {world}\""
      },
      "name": "parserTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-escaped-quotes", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L217-L231", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "parserTool", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}}}}], "raw_output": "{\"name\": \"parserTool\", \"parameters\": {\"text\": \"He said \\\"Hello {world}\\\"\"}}", "output_token_ids": [5018, 609, 794, 330, 9854, 7896, 498, 330, 14105, 794, 5324, 1342, 794, 330, 1548, 1071, 7393, 9906, 314, 14957, 47969, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "parserTool", "arguments": {"text": "He said \"Hello {world}\""}}]}, "tags": ["single-call", "string-escapes", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-vllm-json-without-name-is-content expected_match
Checks that did not pass for llama/l3-vllm-json-without-name-is-content
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected '{"parameters": {}}', got None
expected_matchone failcontent: expected '{"parameters": {}}', got None
expected_matchspecial failcontent: expected '{"parameters": {}}', got None
expected_matchtoken failcontent: expected '{"parameters": {}}', got None
expected_matchrand:1:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:2:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:3:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:4:8 failcontent: expected '{"parameters": {}}', got None
expected_matchrand:5:8 failcontent: expected '{"parameters": {}}', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-json-without-name-is-content --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "{\"parameters\": {}}",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"parameters": {}}

Expected parse

{
  "content": "{\"parameters\": {}}",
  "reasoning_content": null,
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-json-without-name-is-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"parameters\": {}}", "output_token_ids": [5018, 14105, 794, 314, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"parameters\": {}}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. A JSON object without a 'name' is not a call; the text is returned unchanged as content."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l3-vllm-parameters-key expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for llama/l3-vllm-parameters-key
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"query": "test query", "limit": 10}', got ''
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "test query", "limit": 10}', got ''
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"query": "test query", "limit": 10}', got ''
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"query": "test query", "limit": 10}', got ''
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"query": "test query", "limit": 10}'
arguments_jsonone fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] searchTool: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemaone fail[0] searchTool: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] searchTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-parameters-key --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "limit": 10,
-        "query": "test query"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "searchTool"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "limit": 10,
        "query": "test query"
      },
      "name": "searchTool"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L42-L53. Tags: single-call, numeric-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "searchTool", "parameters": {"query": "test query", "limit": 10}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "limit": 10,
        "query": "test query"
      },
      "name": "searchTool"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l3-vllm-parameters-key", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L42-L53", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"parameters\": {\"query\": \"test query\", \"limit\": 10}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 14105, 794, 5324, 1663, 794, 330, 1985, 3319, 498, 330, 9696, 794, 220, 605, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test query", "limit": 10}}]}, "tags": ["single-call", "numeric-arguments", "x-parameters-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128006,
      78191,
      128007,
      271
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 128001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-3.3-70B-Instruct",
    "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "Llama32Detector",
  "tool_call_parser": "llama3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l4-bug-leading-underscore-identifier expected_match
Checks that did not pass for llama/l4-bug-leading-underscore-identifier
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchone failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchspecial failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchtoken failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchrand:1:8 failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchrand:2:8 failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchrand:3:8 failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchrand:4:8 failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []
expected_matchrand:5:8 failcontent: expected None, got '[search(_limit=5)]'; tool_calls: expected ['search'], got []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/imported.jsonl --id llama/l4-bug-leading-underscore-identifier --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 18 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "[search(_limit=5)]",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "_limit": 5
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/56840. Tags: single-call, regression, x-leading-underscore-identifier, x-llama-pythonic.

Raw output

[search(_limit=5)]

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "_limit": 5
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l4-bug-leading-underscore-identifier", "family": "llama", "models": ["meta-llama/Llama-4-Scout-17B-16E-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/56840", "revision": "issue opened 2026-09-14", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's OUTPUTS list."}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"_limit": {"type": "integer"}}}}}], "raw_output": "[search(_limit=5)]", "output_token_ids": [71, 5546, 5810, 22739, 41, 33, 12432], "tokenizer": {"repo": "unsloth/Llama-4-Scout-17B-16E-Instruct", "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"_limit": 5}}]}, "tags": ["single-call", "regression", "x-leading-underscore-identifier", "x-llama-pythonic"], "notes": "Valid Python; JSON-Schema property names may start with '_'. vLLM's non-streaming pythonic regex rejected it and returned the text as content, while streaming produced the call."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "93c6a232af67e2b3bc9218179ec3fea5176ec88b2a4a4fb66e51feb2ef63c0bd",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200005,
      140680,
      200006,
      368
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200001,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<|end_of_text|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-4-Scout-17B-16E-Instruct",
    "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "PythonicDetector",
  "tool_call_parser": "pythonic",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail llama/l4-truncated-missing-close-bracket stream_equals_nonstream
Checks that did not pass for llama/l4-truncated-missing-close-bracket
CheckStrategyResultDetail
stream_equals_nonstreamone failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamspecial failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamtoken failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamrand:1:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamrand:3:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream '[get_weather(city="San Francisco"), get_weather(city="Seattle', got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/llama/recorded.jsonl --id llama/l4-truncated-missing-close-bracket --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/llama/recorded.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "[get_weather(city=\"San Francisco\"), get_weather(city=\"Seattle",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: spec_example, https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama4/prompt_format.md#L204-L207. Tags: truncated, malformed, x-llama-pythonic, x-derived-truncation.

Expected graceful failure: The pythonic call list is cut before its final string, call and list close. (accept: no_tool_calls, content_passthrough, exception).

Raw output

[get_weather(city="San Francisco"), get_weather(city="Seattle
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "llama/l4-truncated-missing-close-bracket", "family": "llama", "models": ["meta-llama/Llama-4-Scout-17B-16E-Instruct"], "spec_version": "0.1", "provenance": {"kind": "spec_example", "source_url": "https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama4/prompt_format.md#L204-L207", "revision": "0e0b8c519242d5833d8c11bffc1232b77ad7f301", "license": "LicenseRef-llama4-community", "generator": "scripts/fixtures/llama/build.py", "attribution": "Llama 4 is licensed under the Llama 4 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. Short example quoted with attribution (THIRD_PARTY_NOTICES.md)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather info for places", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The name of the city to get the weather for"}, "metric": {"type": "string", "description": "The metric for weather. Options are: celsius, fahrenheit", "default": "celsius"}}, "required": ["city"]}}}], "raw_output": "[get_weather(city=\"San Francisco\"), get_weather(city=\"Seattle", "output_token_ids": [193913, 191381, 103469, 508, 33300, 20748, 9811, 872, 191381, 103469, 508, 190860], "tokenizer": {"repo": "unsloth/Llama-4-Scout-17B-16E-Instruct", "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b", "mode": "hf"}, "expected_error": {"reason": "The pythonic call list is cut before its final string, call and list close.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "malformed", "x-llama-pythonic", "x-derived-truncation"], "notes": "Derived, not quoted: the token prefix of llama/l4-meta-pythonic-parallel (a spec_example) without its last token (which carries the closing quote, parenthesis and bracket): the 'missing bracket' failure mode reported in https://github.com/vllm-project/vllm/issues/30722."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "93c6a232af67e2b3bc9218179ec3fea5176ec88b2a4a4fb66e51feb2ef63c0bd",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": null,
  "hf_config_error": "LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.",
  "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200005,
      140680,
      200006,
      368
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "unsloth/Llama-4-Scout-17B-16E-Instruct",
    "revision": "afd8e498c87bda51c7ea8ec68ea2f7c066e6340b",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "PythonicDetector",
  "tool_call_parser": "pythonic",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}