DeepSeek (V3/R1, V3.1, V3.2 DSML, V4 DSML, V4.1 DSML) on llamacpp a25c9865

fail 55% strict pass 35 pass · 29 fail · 21 unsupported

Run

Engine

llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)

Run

2026-09-26T05:36:17Z to 2026-09-26T05:37:20Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}

Results file

data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Fail	Strict pass rate
`expected_match`	33	25	57%
`expected_error`	2	4	33%
`stream_equals_nonstream`	61	3	95%
`no_leakage`	64	0	100%
`arguments_json`	24	4	86%
`arguments_schema`	24	4	86%
`parallel_order`	4	3	57%

29 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail deepseek/sglang-v4-self-closing-invoke expected_match

Checks that did not pass for deepseek/sglang-v4-self-closing-invoke
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/sglang-v4-self-closing-invoke --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "submit"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258. Tags: single-call, empty-arguments, x-self-closing-invoke, x-deepseek-v4.

Raw output

<｜DSML｜tool_calls>
<｜DSML｜invoke name="submit"/>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "submit"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/sglang-v4-self-closing-invoke", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "submit", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"submit\"/>\n</｜DSML｜tool_calls>", "output_token_ids": [30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 35380, 39278, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "submit", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-self-closing-invoke", "x-deepseek-v4"], "notes": "SGLang's test states 'V4 emits <｜DSML｜invoke name=\"x\"/> for zero-arg tools'. The official V4 encoder renders an open/close pair instead, and DeepSeek's reference parser rejects this form; kept because the engine reports the model generating it. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "｜DSML｜",
    "<think>",
    "</think>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V4-Flash"
    }
  },
  "template_identical_to": "models/templates/deepseek-ai-DeepSeek-V4.jinja",
  "template_path": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "7bd4fbdf0c06cd7398ddb6913de6e4a5839b9d06569e74e5b2ec246b375ca9c9",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>",
    "<｜DSML｜tool_calls>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "268450fb8d2629ec4a39f5eba1fbdd290758faf4c4033fb0023ff377607a94ea",
  "vocab_gguf": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "3d8e3933508e59992046e7939bb49885e10875affd40c1cdbba31bcbfca5a4a7"
}

fail deepseek/v3-empty-arguments expected_match

Checks that did not pass for deepseek/v3-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-empty-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, empty-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time
```json
{}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time\n```json\n{}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 20035, 201, 9854, 14479, 201, 76188, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "087e3444fb4f8cb3c090a45e55e6dcb9c26e3c4c39bcf7b3d903f2154627877c",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-marker-in-arguments expected_match

Checks that did not pass for deepseek/v3-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 8 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo
```json
{"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo\n```json\n{\"text\": \"Reply as ```json\\n{\\\"a\\\": 1}\\n``` or use <｜tool▁sep｜> tokens.\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 18843, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 19905, 412, 35739, 14479, 5809, 3757, 4, 67, 55550, 223, 19, 1464, 80, 9854, 469, 1347, 223, 128814, 35977, 2148, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v3"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e09d43a17d3171ed01033951d536c5c566c6c2c1877ecf1f4c7ea47dda349a40",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-nested-json expected_match

Checks that did not pass for deepseek/v3-nested-json
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-nested-json --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,29 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
-      "name": "create_event"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event
```json
{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event\n```json\n{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 10028, 76740, 201, 9854, 14479, 201, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "207eb96ffd4d1f00a7af0ab56e48f350938251a8bf5bc3f66e17c67297c5cc3f",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-numeric-arguments expected_match

Checks that did not pass for deepseek/v3-numeric-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-numeric-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,22 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
-      "name": "calculate"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate
```json
{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate\n```json\n{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 93801, 201, 9854, 14479, 201, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v3"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "c392748030565ca18f28ff94aad01e19dfe18f2dfb8d7b4d9834a7b7845a9041",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-parallel-two-calls expected_match, parallel_order

Checks that did not pass for deepseek/v3-parallel-two-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search'], got 0 []

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-parallel-two-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,26 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Zürich", "unit": "c"}
```<｜tool▁call▁end｜>
<｜tool▁call▁begin｜>function<｜tool▁sep｜>search
```json
{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Zürich\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search\n```json\n{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 201, 128808, 8701, 128814, 14346, 201, 9854, 14479, 201, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 30316, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-single-call expected_match

Checks that did not pass for deepseek/v3-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Paris", "unit": "c"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Paris\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-text-before-call expected_match, stream_equals_nonstream

Checks that did not pass for deepseek/v3-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Let me check the current conditions in London.'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": "Let me check the current conditions in London.",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "London"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"London\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 27571, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v3-unicode-emoji expected_match

Checks that did not pass for deepseek/v3-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-unicode-emoji --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
-      "name": "translate"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate
```json
{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v3-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate\n```json\n{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 77296, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "0dbe48dca9e23b30ebf7f163a1010b7461d14185fefefae927677f77d2d68f36",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/v31-empty-arguments expected_match

Checks that did not pass for deepseek/v31-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-empty-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,10 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, empty-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 20035, 128814, 25902, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{}",
            "name": "get_time"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "087e3444fb4f8cb3c090a45e55e6dcb9c26e3c4c39bcf7b3d903f2154627877c",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-marker-in-arguments expected_match

Checks that did not pass for deepseek/v31-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 8 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
      },
      "name": "echo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{\"text\": \"Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 18843, 128814, 24313, 2067, 3362, 582, 105119, 1277, 223, 128808, 72, 128814, 25902, 477, 92637, 2148, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v31"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"text\": \"Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup.\"}",
            "name": "echo"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "e09d43a17d3171ed01033951d536c5c566c6c2c1877ecf1f4c7ea47dda349a40",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-multi-turn-after-tool-result expected_match

Checks that did not pass for deepseek/v31-multi-turn-after-tool-result
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-multi-turn-after-tool-result --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "café Zürich"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café Zürich"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-multi-turn-after-tool-result", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café Zürich\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 125388, 616, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"query\": \"café Zürich\"}",
            "name": "search"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-nested-json expected_match

Checks that did not pass for deepseek/v31-nested-json
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-nested-json --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 11 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,29 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
-      "name": "create_event"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 10028, 76740, 128814, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}",
            "name": "create_event"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "207eb96ffd4d1f00a7af0ab56e48f350938251a8bf5bc3f66e17c67297c5cc3f",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-numeric-arguments expected_match

Checks that did not pass for deepseek/v31-numeric-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-numeric-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,22 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
-      "name": "calculate"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 93801, 128814, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v31"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}",
            "name": "calculate"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "c392748030565ca18f28ff94aad01e19dfe18f2dfb8d7b4d9834a7b7845a9041",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-parallel-two-calls expected_match, parallel_order

Checks that did not pass for deepseek/v31-parallel-two-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'search'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search'], got 0 []

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-parallel-two-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,26 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Zürich\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 3831, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Zürich\", \"unit\": \"c\"}",
            "name": "get_weather"
          },
          {
            "arguments_raw": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}",
            "name": "search"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-single-call expected_match

Checks that did not pass for deepseek/v31-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Paris", "unit": "c"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Paris\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"Paris\", \"unit\": \"c\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-text-before-call expected_match, stream_equals_nonstream

Checks that did not pass for deepseek/v31-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'Let me check the current conditions in London.'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": "Let me check the current conditions in London.",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "London"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"London\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 27571, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "Let me check the current conditions in London.",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"city\": \"London\"}",
            "name": "get_weather"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "7716fb8476727a5f17f8972a92c7654097007431aac0c586cdac9a95d6a510af",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v31-unicode-emoji expected_match

Checks that did not pass for deepseek/v31-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-unicode-emoji --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
-      "name": "translate"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v31-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 77296, 128814, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}",
            "name": "translate"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "0dbe48dca9e23b30ebf7f163a1010b7461d14185fefefae927677f77d2d68f36",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

fail deepseek/v32-truncated-mid-arguments expected_error, arguments_json, arguments_schema

Checks that did not pass for deepseek/v32-truncated-mid-arguments
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Zür': Unterminated string starting at: line 1 column 9 (char 8))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Zür': Unterminated string starting at: line 1 column 9 (char 8))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-mid-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 9 of fixtures/deepseek/v32.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":\"Zür"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, x-deepseek-v32.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v32-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v32"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v32; history render of the official encoder."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.2@a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜><think></think>",
  "model": "deepseek-ai/DeepSeek-V3.2",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "｜DSML｜",
    "<think>",
    "</think>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "gguf": {
      "available": false,
      "reason": "deepseek-ai--DeepSeek-V3.2.vocab.gguf has no tokenizer.chat_template (llama.cpp would fall back to chatml)"
    }
  },
  "template_identical_to": "models/templates/deepseek-ai-DeepSeek-V3.2.jinja",
  "template_path": "models/templates/deepseek-ai-DeepSeek-V3.2.jinja",
  "template_sha256": "8e5a13fcc4bdd4d943311fc59064b805770fdeddfccbd7f1c252cc16e8acaf3d",
  "template_source": "llamacpp",
  "template_source_reason": "deepseek-ai--DeepSeek-V3.2.vocab.gguf has no tokenizer.chat_template (llama.cpp would fall back to chatml)",
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>",
    "<｜DSML｜function_calls>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "deepseek-ai/DeepSeek-V3.2@a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.2.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": [
    "tokenizer_config.json: add_bos_token=true"
  ],
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c5ec56f20d6c055dfa0c24d82a58bd2319f6db8d54949d0a5dd17179370c8d3b"
}

fail deepseek/v32-truncated-second-parallel-call expected_match, arguments_json, arguments_schema

Checks that did not pass for deepseek/v32-truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`nonstream`	fail	[1] search: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_json`	`token`	fail	[1] search: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`nonstream`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[1] search: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-second-parallel-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 16 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{"
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v32-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v32"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v32; history render of the official encoder."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.2@a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜><think></think>",
  "model": "deepseek-ai/DeepSeek-V3.2",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "｜DSML｜",
    "<think>",
    "</think>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "gguf": {
      "available": false,
      "reason": "deepseek-ai--DeepSeek-V3.2.vocab.gguf has no tokenizer.chat_template (llama.cpp would fall back to chatml)"
    }
  },
  "template_identical_to": "models/templates/deepseek-ai-DeepSeek-V3.2.jinja",
  "template_path": "models/templates/deepseek-ai-DeepSeek-V3.2.jinja",
  "template_sha256": "8e5a13fcc4bdd4d943311fc59064b805770fdeddfccbd7f1c252cc16e8acaf3d",
  "template_source": "llamacpp",
  "template_source_reason": "deepseek-ai--DeepSeek-V3.2.vocab.gguf has no tokenizer.chat_template (llama.cpp would fall back to chatml)",
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>",
    "<｜DSML｜function_calls>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "deepseek-ai/DeepSeek-V3.2@a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.2.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": [
    "tokenizer_config.json: add_bos_token=true"
  ],
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "c5ec56f20d6c055dfa0c24d82a58bd2319f6db8d54949d0a5dd17179370c8d3b"
}

fail deepseek/v4-truncated-mid-arguments expected_error, arguments_json, arguments_schema

Checks that did not pass for deepseek/v4-truncated-mid-arguments
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
`expected_error`	`token`	fail	returned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
`arguments_json`	`nonstream`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Zür': Unterminated string starting at: line 1 column 9 (char 8))
`arguments_json`	`token`	fail	[0] get_weather: arguments are not valid JSON text ('{"city":"Zür': Unterminated string starting at: line 1 column 9 (char 8))
`arguments_schema`	`nonstream`	fail	[0] get_weather: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-mid-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 9 of fixtures/deepseek/v4.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":\"Zür"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, x-deepseek-v4.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v4-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v4"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v4; history render of the official encoder."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "｜DSML｜",
    "<think>",
    "</think>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V4-Flash"
    }
  },
  "template_identical_to": "models/templates/deepseek-ai-DeepSeek-V4.jinja",
  "template_path": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "7bd4fbdf0c06cd7398ddb6913de6e4a5839b9d06569e74e5b2ec246b375ca9c9",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>",
    "<｜DSML｜tool_calls>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "3d8e3933508e59992046e7939bb49885e10875affd40c1cdbba31bcbfca5a4a7"
}

fail deepseek/v4-truncated-second-parallel-call expected_match, arguments_json, arguments_schema

Checks that did not pass for deepseek/v4-truncated-second-parallel-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got ['get_weather', 'search']
`arguments_json`	`nonstream`	fail	[1] search: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_json`	`token`	fail	[1] search: arguments are not valid JSON text ('{': Expecting property name enclosed in double quotes: line 1 column 2 (char 1))
`arguments_schema`	`nonstream`	fail	[1] search: arguments are not a JSON object; not validated
`arguments_schema`	`token`	fail	[1] search: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-second-parallel-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 17 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,10 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{"
+      },
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/v4-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v4"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v4; history render of the official encoder."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": true,
  "enable_thinking": false,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "｜DSML｜",
    "<think>",
    "</think>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V4-Flash"
    }
  },
  "template_identical_to": "models/templates/deepseek-ai-DeepSeek-V4.jinja",
  "template_path": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "7bd4fbdf0c06cd7398ddb6913de6e4a5839b9d06569e74e5b2ec246b375ca9c9",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "</think>",
    "<｜DSML｜tool_calls>"
  ],
  "thinking_start_tag": "<think>",
  "tokenizer": "deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "2309009459b4475c5ddada39a14bc9114e87e46a4469d34c5c361ffa5a0badaf",
  "vocab_gguf": "deepseek-ai--DeepSeek-V4-Flash.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "3d8e3933508e59992046e7939bb49885e10875affd40c1cdbba31bcbfca5a4a7"
}

fail deepseek/vllm-v3-escaped-strings expected_match

Checks that did not pass for deepseek/vllm-v3-escaped-strings
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-escaped-strings --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 5 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,14 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "newline": "line1\nline2",
-        "path": "C:\\Users\\file",
-        "text": "He said \"hello\""
-      },
-      "name": "send_message"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61. Tags: single-call, string-escapes, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message
```json
{"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "newline": "line1\nline2",
        "path": "C:\\Users\\file",
        "text": "He said \"hello\""
      },
      "name": "send_message"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-escaped-strings", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "send_message", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "path": {"type": "string"}, "newline": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message\n```json\n{\"text\": \"He said \\\"hello\\\"\", \"path\": \"C:\\\\Users\\\\file\", \"newline\": \"line1\\nline2\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 35397, 53735, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 3158, 1359, 36952, 33310, 28609, 1760, 582, 9860, 3362, 582, 37, 111396, 24979, 7387, 8104, 1760, 582, 2839, 1836, 3362, 582, 1836, 19, 5809, 1836, 20, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "send_message", "arguments": {"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}}]}, "tags": ["single-call", "string-escapes", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "0b567a2bbda00de140e0190cee46cf563486a954ee445cebb2d1786f2fb035a7",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v3-malformed-missing-brace expected_error

Checks that did not pass for deepseek/vllm-v3-malformed-missing-brace
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
`expected_error`	`token`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/malformed.jsonl --id deepseek/vllm-v3-malformed-missing-brace --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 1 of fixtures/deepseek/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68. Tags: malformed, x-deepseek-v3.

Expected graceful failure: The arguments JSON is missing its closing brace; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo"
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-malformed-missing-brace", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 2701, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "The arguments JSON is missing its closing brace; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "x-deepseek-v3"], "notes": "malformed_input_outputs[0] of the test. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "83a43c9669f70c80609c321c8bd29d69ac1e6d718f98d32c0326c5effc818443",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v3-malformed-missing-call-tokens expected_error

Checks that did not pass for deepseek/vllm-v3-malformed-missing-call-tokens
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
`expected_error`	`token`	fail	outcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/malformed.jsonl --id deepseek/vllm-v3-malformed-missing-call-tokens --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/deepseek/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68. Tags: malformed, x-deepseek-v3.

Expected graceful failure: <｜tool▁call▁begin｜>/<｜tool▁call▁end｜> are missing inside the calls section; no well-formed call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<｜tool▁calls▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo"}
```<｜tool▁calls▁end｜>

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-malformed-missing-call-tokens", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"}\n```<｜tool▁calls▁end｜>", "output_token_ids": [128806, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 60676, 9854, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "<｜tool▁call▁begin｜>/<｜tool▁call▁end｜> are missing inside the calls section; no well-formed call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "x-deepseek-v3"], "notes": "malformed_input_outputs[1] of the test. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "83a43c9669f70c80609c321c8bd29d69ac1e6d718f98d32c0326c5effc818443",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v3-parallel-no-newline expected_match, parallel_order

Checks that did not pass for deepseek/vllm-v3-parallel-no-newline
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'search_hotels'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'search_hotels'], got 0 []

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-parallel-no-newline --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 3 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,20 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
-      "name": "search_hotels"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels
```json
{"location": "Tokyo", "check_in": "2025-01-15"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "check_in": "2025-01-15",
        "location": "Tokyo"
      },
      "name": "search_hotels"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-parallel-no-newline", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L31", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}, {"type": "function", "function": {"name": "search_hotels", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "check_in": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels\n```json\n{\"location\": \"Tokyo\", \"check_in\": \"2025-01-15\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128808, 8701, 128814, 14346, 17462, 347, 1713, 201, 9854, 14479, 201, 24313, 33182, 3362, 582, 106239, 1760, 582, 9547, 12743, 3362, 582, 939, 23, 15, 2616, 15, 856, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}, {"name": "search_hotels", "arguments": {"location": "Tokyo", "check_in": "2025-01-15"}}]}, "tags": ["parallel-calls", "x-deepseek-v3"], "notes": "Calls are adjacent here; the official template separates them with '\\n'. output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "89d0117d83e1652075a801dee7803d0ebccddf50e1d6db85e9ccdce2bb1985b0",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v3-single-call expected_match

Checks that did not pass for deepseek/vllm-v3-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-single-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 2 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L27", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "83a43c9669f70c80609c321c8bd29d69ac1e6d718f98d32c0326c5effc818443",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v3-various-data-types expected_match

Checks that did not pass for deepseek/vllm-v3-various-data-types
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-various-data-types --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 4 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,26 +1,6 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "array_field": [
-          "a",
-          "b",
-          "c"
-        ],
-        "bool_field": true,
-        "empty_array": [],
-        "empty_object": {},
-        "float_field": 3.14,
-        "int_field": 42,
-        "null_field": null,
-        "object_field": {
-          "nested": "value"
-        },
-        "string_field": "hello"
-      },
-      "name": "test_function"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39. Tags: single-call, numeric-arguments, nested-json, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function
```json
{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "array_field": [
          "a",
          "b",
          "c"
        ],
        "bool_field": true,
        "empty_array": [],
        "empty_object": {},
        "float_field": 3.14,
        "int_field": 42,
        "null_field": null,
        "object_field": {
          "nested": "value"
        },
        "string_field": "hello"
      },
      "name": "test_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v3-various-data-types", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "test_function", "parameters": {"type": "object", "properties": {"string_field": {"type": "string"}, "int_field": {"type": "integer"}, "float_field": {"type": "number"}, "bool_field": {"type": "boolean"}, "null_field": {"type": "null"}, "array_field": {"type": "array"}, "object_field": {"type": "object"}, "empty_array": {"type": "array"}, "empty_object": {"type": "object"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function\n```json\n{\"string_field\": \"hello\", \"int_field\": 42, \"float_field\": 3.14, \"bool_field\": true, \"null_field\": null, \"array_field\": [\"a\", \"b\", \"c\"], \"object_field\": {\"nested\": \"value\"}, \"empty_array\": [], \"empty_object\": {}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 7958, 64334, 201, 9854, 14479, 201, 24313, 4463, 55291, 3362, 582, 33310, 1760, 582, 650, 55291, 3362, 223, 3180, 14, 582, 15891, 55291, 3362, 223, 21, 16, 929, 14, 582, 20600, 55291, 3362, 3292, 14, 582, 16042, 55291, 3362, 5100, 14, 582, 7386, 55291, 3362, 20584, 67, 1760, 582, 68, 1760, 582, 69, 45390, 582, 10325, 55291, 3362, 28612, 80, 12904, 3362, 582, 5197, 87928, 582, 15566, 30549, 3362, 66517, 582, 15566, 55774, 3362, 26102, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "test_function", "arguments": {"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}}]}, "tags": ["single-call", "numeric-arguments", "nested-json", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "format": "peg-native",
  "generation_prompt": "",
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "function<｜tool▁sep｜>",
    "```json\n{'AA_ARG_FST_AA': 'XXXX', 'BB_ARG_SND_BB': 'YYYY'}\n```",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'",
    "': 'XXXX'}\n```<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>FFF_FIRST_FUN_F\n```json\n{'AA_ARG_FST_AA': '",
    "'}\n```"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after deepseek-ai/DeepSeek-V3-0324"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3-0324@e9b33add76883f293d6bf61f6bd89b497e80e335",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "6f61e4f1161682269b410bbcb8322083bdd4364dcdbc9f5d0873df859fab9461",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3-0324.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "ef0dce472db9af93ffb8a1e72bb8eeaf70927508bb587d90d196260867eff6c4"
}

fail deepseek/vllm-v31-text-before-call expected_match, stream_equals_nonstream

Checks that did not pass for deepseek/vllm-v31-text-before-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`expected_match`	`token`	fail	parser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got 'normal text'

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v31-text-before-call --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
-  "content": "normal text",
+  "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "foo"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,6 @@
 {
   "content": "normal text",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "foo"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25. Tags: single-call, text-before-call, x-deepseek-v31.

Raw output

normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{"x":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "normal text",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "foo"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "deepseek/vllm-v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "foo", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}}}}}], "raw_output": "normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{\"x\":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [13644, 3051, 128806, 128808, 40897, 128814, 24313, 90, 3362, 19, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "normal text", "reasoning_content": null, "tool_calls": [{"name": "foo", "arguments": {"x": 1}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}

Parser configuration

{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenized_matches_raw_output": false,
  "enable_thinking": false,
  "end_token": "<｜end▁of▁sentence｜>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'DeepSeek V3.2/V4' for V3.2/V4 templates, autoparser otherwise",
  "fixture_generation_prompt": "<｜Assistant｜></think>",
  "fixture_tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "format": "peg-native",
  "generation_prompt": "<｜Assistant｜></think>",
  "model": "deepseek-ai/DeepSeek-V3.1",
  "parallel_tool_calls": true,
  "preserved_tokens": [
    "<｜tool▁calls▁begin｜>",
    "<｜tool▁calls▁end｜>",
    "<｜tool▁call▁begin｜>",
    "<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'",
    "': 'XXXX'}<｜tool▁call▁end｜>",
    "<｜tool▁sep｜>{'AA_ARG_FST_AA': '",
    "'}"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": true,
      "format": "peg-native",
      "nonstream": {
        "content": "normal text",
        "exception": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "arguments_raw": "{\"x\":1}",
            "name": "foo"
          }
        ]
      },
      "path": "models/templates/deepseek-ai-DeepSeek-V3.1.jinja",
      "sha256": "d9f5f351b276cf9e81d009db08830db7d38e94927aecd09a7beb43e2aafee94e"
    }
  },
  "template_identical_to": null,
  "template_path": "deepseek-ai--DeepSeek-V3.1.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": false,
  "template_variant": "default",
  "thinking_end_tags": [],
  "thinking_start_tag": "",
  "tokenizer": "deepseek-ai/DeepSeek-V3.1@c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "cf0da75a0bdb2580ca8b02be125dd094bc1fc1e9f8e3547714553ccd55cc5de0",
  "vocab_gguf": "deepseek-ai--DeepSeek-V3.1.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "346cae28cfa6863405f8e2fecdc8c6f101249fdd9cf7bc8f235bb87bf40009e7"
}

Unsupported fixtures

The adapter declined these fixtures because this engine version has no parser for the family or model.

deepseek/v41-empty-arguments: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-long-arguments: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-marker-in-arguments: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-multi-turn-after-tool-result: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-no-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-numeric-arguments: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-parallel-two-calls: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-reasoning-no-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-reasoning-parallel-text-before: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-reasoning-single-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-single-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-string-false-json: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-text-before-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-truncated-in-reasoning: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-truncated-mid-arguments: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-truncated-second-parallel-call: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-unescaped-string-value: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-unicode-emoji: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/v41-whitespace-string-value: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/vllm-v41-reasoning-text-before-parallel-calls: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported
deepseek/vllm-v41-text-before-parallel-calls: llama.cpp convert_hf_to_gguf.py cannot convert deepseek-ai/DeepSeek-V4.1-Flash: ERROR:hf-to-gguf:Model DeepseekV41ForCausalLM is not supported

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model