Llama (3.1, 3.2, 3.3, 4) on llamacpp a25c9865

fail 58% strict pass 14 pass · 10 fail · 10 unsupported

Run

Engine

llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)

Run

2026-09-26T05:36:17Z to 2026-09-26T05:37:20Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}

Results file

data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Fail	Strict pass rate
`expected_match`	14	9	61%
`expected_error`	0	1	0%
`stream_equals_nonstream`	20	4	83%
`no_leakage`	24	0	100%
`arguments_json`	14	5	74%
`arguments_schema`	14	5	74%

10 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail llama/l3-bug-empty-json-object-in-content expected_match

Checks that did not pass for llama/l3-bug-empty-json-object-in-content
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-empty-json-object-in-content --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 16 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
-  "content": "{}",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{}

Expected parse

{
  "content": "{}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-bug-empty-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue: input '{}' -> content '' (the message is gone)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{}", "output_token_ids": [6390], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"]}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "f3b4dbb6b51666a65bdc9a93d59a4df4a56b6bc27de5a67d1f183405fec1b9a6",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-bug-leading-json-object-in-content expected_match

Checks that did not pass for llama/l3-bug-leading-json-object-in-content
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-leading-json-object-in-content --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 15 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
-  "content": "{\"a\": 1} is a dict",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: bug_report, https://github.com/sgl-project/sglang/issues/35562. Tags: no-call, regression, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"a": 1} is a dict

Expected parse

{
  "content": "{\"a\": 1} is a dict",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-bug-leading-json-object-in-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/sgl-project/sglang/issues/35562", "revision": "issue opened 2026-08-19", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's reproduction (Llama-3.2-1B-Instruct, tools=[get_weather])."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {}}}], "raw_output": "{\"a\": 1} is a dict", "output_token_ids": [5018, 64, 794, 220, 16, 92, 374, 264, 6587], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"a\": 1} is a dict", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "regression", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "Not a call (no name/parameters). SGLang's llama3 parser deleted the leading JSON and returned 'is a dict'. The issue's expectation: when no call is found the text must be unchanged."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "f3b4dbb6b51666a65bdc9a93d59a4df4a56b6bc27de5a67d1f183405fec1b9a6",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-bug-whole-call-single-delta expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for llama/l3-bug-whole-call-single-delta
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-bug-whole-call-single-delta --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 17 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,9 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Tokyo"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_weather"

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/48294. Tags: single-call, regression, x-single-delta, x-arguments-key, x-llama-json.

Raw output

{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-bug-whole-call-single-delta", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/48294", "revision": "issue opened 2026-07-11", "license": "NOASSERTION", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Quoted from the issue's minimal reproduction for llama3_json."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "regression", "x-single-delta", "x-arguments-key", "x-llama-json"], "notes": "vLLM's llama3_json streaming path emitted nothing when this whole message arrived in one delta (the 'one' chunking strategy replays exactly that)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e38cb2d2622e79af3c7fe7b045e93921f14d2098832c30d21a54818c79a0e455",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-json-answer-not-a-call expected_match

Checks that did not pass for llama/l3-json-answer-not-a-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-json-answer-not-a-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
-  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"capital": "Paris", "country": "France"}

Expected parse

{
  "content": "{\"capital\": \"Paris\", \"country\": \"France\"}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-json-answer-not-a-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"capital\": \"Paris\", \"country\": \"France\"}", "output_token_ids": [5018, 66163, 794, 330, 60704, 498, 330, 11389, 794, 330, 50100, 9388], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"capital\": \"Paris\", \"country\": \"France\"}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "A plain JSON answer: it has no name/parameters keys, so it is content, not a call (cf. https://github.com/sgl-project/sglang/issues/35562)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e7dfca0e9459a4599659caea2fe7ef822e0e304dffe34cb0793309e8f3062f5c",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-marker-in-arguments expected_match

Checks that did not pass for llama/l3-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <\|python_tag\|>; headers look like <\|start_header_id\|>ipython<\|end_header_id\|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'
`expected_match`	`token`	fail	tool_calls[0].arguments: expected '{"path": "docs/llama.md", "content": "Tool calls may start with <\|python_tag\|>; headers look like <\|start_header_id\|>ipython<\|end_header_id\|>."}', got '{"path": "docs/llama.md", "content": "Tool calls may start with ; headers look like ipython."}'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/l3-render.jsonl --id llama/l3-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/llama/l3-render.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
+        "content": "Tool calls may start with ; headers look like ipython.",
         "path": "docs/llama.md"
       },

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: single-call, marker-in-arguments, x-parameters-key, x-llama-json.

Raw output

{"name": "write_file", "parameters": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.",
        "path": "docs/llama.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-marker-in-arguments", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct", "meta-llama/Llama-3.1-8B-Instruct", "meta-llama/Llama-3.2-3B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", "attribution": "Official Llama 3.3 template via the unsloth mirror (byte-identical to llama.cpp's copy at a25c9865fe03c954c93fd755b5d79ae86ba99750); meta-llama repos are gated."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"write_file\", \"parameters\": {\"path\": \"docs/llama.md\", \"content\": \"Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>.\"}}", "output_token_ids": [5018, 609, 794, 330, 5040, 2517, 498, 330, 14105, 794, 5324, 2398, 794, 330, 14452, 14, 657, 3105, 22030, 498, 330, 1834, 794, 330, 7896, 6880, 1253, 1212, 449, 220, 128010, 26, 7247, 1427, 1093, 220, 128006, 23799, 4690, 128007, 1210, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "docs/llama.md", "content": "Tool calls may start with <|python_tag|>; headers look like <|start_header_id|>ipython<|end_header_id|>."}}]}, "tags": ["single-call", "marker-in-arguments", "x-parameters-key", "x-llama-json"], "notes": "The JSON string contains <|python_tag|> and header tokens, tokenized as the special tokens. A parser that splits on <|python_tag|> must not cut inside the string."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"path\": \"docs/llama.md\", \"content\": \"Tool calls may start with ; headers look like ipython.\"}",
            "name": "write_file"
          }
        ]
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e7dfca0e9459a4599659caea2fe7ef822e0e304dffe34cb0793309e8f3062f5c",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-sglang-python-tag-call expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for llama/l3-sglang-python-tag-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-python-tag-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Beijing"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,9 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Beijing"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_weather"

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76. Tags: single-call, x-python-tag-prefix, x-arguments-key, x-llama-json.

Raw output

<|python_tag|>{"name": "get_weather", "arguments": {"city": "Beijing"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Beijing"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-sglang-python-tag-call", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L70-L76", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "<|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Beijing\"}}", "output_token_ids": [128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 3513, 24570, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Beijing"}}]}, "tags": ["single-call", "x-python-tag-prefix", "x-arguments-key", "x-llama-json"]}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e38cb2d2622e79af3c7fe7b045e93921f14d2098832c30d21a54818c79a0e455",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-sglang-text-before-python-tag expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for llama/l3-sglang-text-before-python-tag
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Let me check. '; tool_calls: nonstream [], got ['get_weather']
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-sglang-text-before-python-tag --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 8 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": "Let me check. ",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, rand:1:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": "Let me check. ",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:2:8, rand:3:8

--- expected
+++ observed
@@ -1,9 +1,10 @@
 {
   "content": "Let me check. ",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "Tokyo"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "get_weather"

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88. Tags: single-call, text-before-call, x-python-tag-prefix, x-arguments-key, x-text-plus-call, x-llama-json.

Raw output

Let me check. <|python_tag|>{"name": "get_weather", "arguments": {"city": "Tokyo"}}

Expected parse

{
  "content": "Let me check. ",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-sglang-text-before-python-tag", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_llama32_detector.py#L84-L88", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright SGLang contributors (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather information", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query"}}, "required": ["query"]}}}], "raw_output": "Let me check. <|python_tag|>{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}", "output_token_ids": [10267, 757, 1817, 13, 220, 128010, 5018, 609, 794, 330, 456, 70464, 498, 330, 16774, 794, 5324, 9103, 794, 330, 53954, 16417, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "Let me check. ", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo"}}]}, "tags": ["single-call", "text-before-call", "x-python-tag-prefix", "x-arguments-key", "x-text-plus-call", "x-llama-json"], "notes": "SGLang asserts the normal text keeps its trailing space."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e38cb2d2622e79af3c7fe7b045e93921f14d2098832c30d21a54818c79a0e455",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-truncated-inside-parameters expected_error, arguments_json, arguments_schema

Checks that did not pass for llama/l3-truncated-inside-parameters
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
`expected_error`	`token`	fail	returned 1 tool call(s) ['create_event'] for max_tokens hit inside the parameters object; the JSON never closes.
`arguments_json`	`nonstream`	fail	[0] create_event: arguments are not valid JSON text ('{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,': Expecting property name enclosed in double quotes: line 1 column 97 (char 96))
`arguments_json`	`token`	fail	[0] create_event: arguments are not valid JSON text ('{"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,': Expecting property name enclosed in double quotes: line 1 column 97 (char 96))
`arguments_schema`	`nonstream`	fail	[0] create_event: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] create_event: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/truncated.jsonl --id llama/l3-truncated-inside-parameters --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/llama/truncated.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,"
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja. Tags: truncated, x-llama-json.

Expected graceful failure: max_tokens hit inside the parameters object; the JSON never closes. (accept: no_tool_calls, content_passthrough, exception).

Raw output

{"name": "create_event", "parameters": {"title": "Standup", "attendees": ["ana@example.com", "bo@example.com"], "duration_minutes": 15,

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-truncated-inside-parameters", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/99cd0d2c829e92a67c844f9144c2509632e5c87f/chat_template.jinja", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "license": "LicenseRef-llama3.3-community", "generator": "scripts/fixtures/llama/build.py", "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search_web", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}}}}, {"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "attendees": {"type": "array", "items": {"type": "string"}}, "duration_minutes": {"type": "integer"}, "all_day": {"type": "boolean"}, "reminder_minutes": {"type": ["integer", "null"]}}, "required": ["title"]}}}, {"type": "function", "function": {"name": "update_settings", "description": "Update user settings.", "parameters": {"type": "object", "properties": {"settings": {"type": "object"}, "dry_run": {"type": "boolean"}}, "required": ["settings"]}}}, {"type": "function", "function": {"name": "convert_units", "description": "Convert a value between units.", "parameters": {"type": "object", "properties": {"value": {"type": "number"}, "from_unit": {"type": "string"}, "to_unit": {"type": "string"}}, "required": ["value", "from_unit", "to_unit"]}}}], "raw_output": "{\"name\": \"create_event\", \"parameters\": {\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,", "output_token_ids": [5018, 609, 794, 330, 3261, 6891, 498, 330, 14105, 794, 5324, 2150, 794, 330, 49833, 455, 498, 330, 57558, 5633, 794, 4482, 3444, 36587, 916, 498, 330, 754, 36587, 916, 8073, 330, 17456, 68755, 794, 220, 868, 11], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected_error": {"reason": "max_tokens hit inside the parameters object; the JSON never closes.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-llama-json"], "notes": "Token prefix of llama/l3-typed-values."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"title\": \"Standup\", \"attendees\": [\"ana@example.com\", \"bo@example.com\"], \"duration_minutes\": 15,",
            "name": "create_event"
          }
        ]
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e7dfca0e9459a4599659caea2fe7ef822e0e304dffe34cb0793309e8f3062f5c",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-vllm-arguments-key expected_match, stream_equals_nonstream, arguments_json, arguments_schema

Checks that did not pass for llama/l3-vllm-arguments-key
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	tool_calls: nonstream [], got ['searchTool']
`arguments_json`	`token`	fail	[0] searchTool: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`token`	fail	[0] searchTool: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-arguments-key --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "test"
-      },
-      "name": "searchTool"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:3:8

--- expected
+++ observed
@@ -1,9 +1,10 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "query": "test"
+        "<arguments_raw, not valid JSON>": "{"
       },
       "name": "searchTool"

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84. Tags: single-call, x-arguments-key, x-llama-json.

Raw output

{"name": "searchTool", "arguments": {"query": "test"}}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "test"
      },
      "name": "searchTool"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-vllm-arguments-key", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L76-L84", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"name\": \"searchTool\", \"arguments\": {\"query\": \"test\"}}", "output_token_ids": [5018, 609, 794, 330, 1874, 7896, 498, 330, 16774, 794, 5324, 1663, 794, 330, 1985, 32075], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "searchTool", "arguments": {"query": "test"}}]}, "tags": ["single-call", "x-arguments-key", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. Llama's documented key is 'parameters'; models also emit 'arguments', which vLLM and SGLang accept."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "694594bd6258bb885f7124fb722ca65a57cd4e09cf65dccad72b55d00bfba6d9",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

fail llama/l3-vllm-json-without-name-is-content expected_match

Checks that did not pass for llama/l3-vllm-json-without-name-is-content
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/llama/imported.jsonl --id llama/l3-vllm-json-without-name-is-content --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/llama/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
-  "content": "{\"parameters\": {}}",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241. Tags: no-call, x-json-in-content-not-a-call, x-llama-json.

Raw output

{"parameters": {}}

Expected parse

{
  "content": "{\"parameters\": {}}",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "llama/l3-vllm-json-without-name-is-content", "family": "llama", "models": ["meta-llama/Llama-3.3-70B-Instruct"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_llama3_json_tool_parser.py#L234-L241", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/llama/imported.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)"}, "tools": [{"type": "function", "function": {"name": "searchTool", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "{\"parameters\": {}}", "output_token_ids": [5018, 14105, 794, 314, 3500], "tokenizer": {"repo": "unsloth/Llama-3.3-70B-Instruct", "revision": "99cd0d2c829e92a67c844f9144c2509632e5c87f", "mode": "hf"}, "expected": {"content": "{\"parameters\": {}}", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-json-in-content-not-a-call", "x-llama-json"], "notes": "The engine test runs without a tools list; the tool schemas here are minimal stand-ins. A JSON object without a 'name' is not a call; the text is returned unchanged as content."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<|end_of_text|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "autoparser (from the chat template)",
  "fixture_generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "fixture_tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "format": "peg-native",
  "generation_prompt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "parallel_tool_calls": false,
  "preserved_tokens": [],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": null,
        "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
        "reasoning_content": null,
        "tool_calls": []
      },
      "path": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
      "sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65"
    }
  },
  "template_identical_to": "models/templates/meta-llama-Llama-3.3-70B-Instruct.jinja",
  "template_path": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "unsloth/Llama-3.3-70B-Instruct@99cd0d2c829e92a67c844f9144c2509632e5c87f",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "694594bd6258bb885f7124fb722ca65a57cd4e09cf65dccad72b55d00bfba6d9",
  "vocab_gguf": "unsloth--Llama-3.3-70B-Instruct.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "0846cc67bf522638d67375353e22cf64c38b61487dfff0a877e6ce245e340ae8"
}

Unsupported fixtures

The adapter declined these fixtures because this engine version has no parser for the family or model.

llama/l4-bug-leading-underscore-identifier: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-meta-pythonic-int-and-string: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-meta-pythonic-parallel: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-truncated-missing-close-bracket: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-escaped-strings: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-more-types: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-no-call: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-parallel-no-spaces: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-parameterless: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template
llama/l4-vllm-pythonic-python-start-end: no authoritative Llama 4 chat template: meta-llama repos are gated and the ungated mirror's template is not Meta's; llama.cpp derives its parser from the template

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model