CanIToolCall caniuse.com for tool calling

DeepSeek (V3/R1, V3.1, V3.2 DSML, V4 DSML, V4.1 DSML) on vllm 0.30.0

fail 51% strict pass 43 pass · 9 soft pass · 33 fail

Run

Engine
vllm 0.30.0
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}
Results file
data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 43 9 26 0 55%
expected_error 1 0 6 0 14%
stream_equals_nonstream 56 0 29 0 66%
split_invariance 60 0 25 0 71%
no_leakage 83 0 2 0 98%
arguments_json 54 0 18 0 75%
arguments_schema 50 0 22 0 69%
parallel_order 8 0 3 0 73%

Fixtures needing attention

fail deepseek/sglang-v4-self-closing-invoke expected_match, no_leakage, arguments_schema
Checks that did not pass for deepseek/sglang-v4-self-closing-invoke
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchone failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchspecial failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchtoken failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchrand:1:8 failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchrand:2:8 failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchrand:3:8 failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchrand:4:8 failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
expected_matchrand:5:8 failtool_calls: expected ['submit'], got ['submit"/>\n</｜DSML｜tool_calls>']
no_leakagenonstream failtool_calls[0].name contains '｜DSML｜'
no_leakageone failtool_calls[0].name contains '｜DSML｜'
no_leakagespecial failtool_calls[0].name contains '｜DSML｜'
no_leakagetoken failtool_calls[0].name contains '｜DSML｜'
no_leakagerand:1:8 failtool_calls[0].name contains '｜DSML｜'
no_leakagerand:2:8 failtool_calls[0].name contains '｜DSML｜'
no_leakagerand:3:8 failtool_calls[0].name contains '｜DSML｜'
no_leakagerand:4:8 failtool_calls[0].name contains '｜DSML｜'
no_leakagerand:5:8 failtool_calls[0].name contains '｜DSML｜'
arguments_schemanonstream fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemaone fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemaspecial fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schematoken fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemarand:1:8 fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemarand:2:8 fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemarand:3:8 fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemarand:4:8 fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool
arguments_schemarand:5:8 fail[0] 'submit"/>\n</｜DSML｜tool_calls>' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/sglang-v4-self-closing-invoke --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {},
-      "name": "submit"
+      "name": "submit\"/>\n</｜DSML｜tool_calls>"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258. Tags: single-call, empty-arguments, x-self-closing-invoke, x-deepseek-v4.

Raw output

<｜DSML｜tool_calls>
<｜DSML｜invoke name="submit"/>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "submit"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/sglang-v4-self-closing-invoke", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "submit", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"submit\"/>\n</｜DSML｜tool_calls>", "output_token_ids": [30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 35380, 39278, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "submit", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-self-closing-invoke", "x-deepseek-v4"], "notes": "SGLang's test states 'V4 emits <｜DSML｜invoke name=\"x\"/> for zero-arg tools'. The official V4 encoder renders an open/close pair instead, and DeepSeek's reference parser rejects this form; kept because the engine reports the model generating it. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "model_rule": "DeepSeek-V4",
  "model_type": "deepseek_v4",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "bfbb51e563b89c7d3581864cd0ea4d975b96e14fed034a222caf9d50c549b99d",
    "source": "vLLM tokenizer_mode=deepseek_v4 encoder (vllm.tokenizers.deepseek_v4_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v4",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "requested_mode": "auto",
    "resolved_revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v4",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v3-empty-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_time'], got []
expected_matchspecial failtool_calls: expected ['get_time'], got []
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{}', got ''
stream_equals_nonstreamone failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{}', got ''
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_time'] | rand:1:8 (fail): tool_calls: one [], got ['get_time'] | rand:2:8 (fail): tool_calls: one [], got ['get_time'] | rand:3:8 (fail): tool_calls: one [], got ['get_time'] | rand:4:8 (fail): tool_calls: one [], got ['get_time'] | rand:5:8 (fail): tool_calls: one [], got ['get_time']
arguments_jsonrand:2:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemarand:2:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-empty-arguments --strategy one --strategy rand:2:8 --strategy rand:3:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:2:8, rand:3:8

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token, rand:1:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, empty-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time
```json
{}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time\n```json\n{}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 20035, 201, 9854, 14479, 201, 76188, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v3-marker-in-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['echo'], got []
expected_matchspecial failtool_calls: expected ['echo'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['echo'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['echo'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['echo'] | rand:1:8 (fail): tool_calls: one [], got ['echo'] | rand:2:8 (fail): tool_calls: one [], got ['echo'] | rand:3:8 (fail): tool_calls: one [], got ['echo'] | rand:4:8 (fail): tool_calls: one [], got ['echo'] | rand:5:8 (fail): tool_calls: one [], got ['echo']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-marker-in-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, marker-in-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo
```json
{"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo\n```json\n{\"text\": \"Reply as ```json\\n{\\\"a\\\": 1}\\n``` or use <｜tool▁sep｜> tokens.\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 18843, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 19905, 412, 35739, 14479, 5809, 3757, 4, 67, 55550, 223, 19, 1464, 80, 9854, 469, 1347, 223, 128814, 35977, 2148, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v3"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-nested-json expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v3-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['create_event'], got []
expected_matchspecial failtool_calls: expected ['create_event'], got []
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
stream_equals_nonstreamone failtool_calls: nonstream ['create_event'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['create_event'], got []
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
split_invariance* failvs one: token (fail): tool_calls: one [], got ['create_event'] | rand:1:8 (fail): tool_calls: one [], got ['create_event'] | rand:2:8 (fail): tool_calls: one [], got ['create_event'] | rand:3:8 (fail): tool_calls: one [], got ['create_event'] | rand:4:8 (fail): tool_calls: one [], got ['create_event'] | rand:5:8 (fail): tool_calls: one [], got ['create_event']
arguments_jsonrand:2:8 fail[0] create_event: arguments are not valid JSON text ('{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...: Unterminated string starting at: line 1 column 195 (char 194))
arguments_schemarand:2:8 fail[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-nested-json --strategy one --strategy rand:2:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,28 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
-      "name": "create_event"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:2:8

--- expected
+++ observed
@@ -5,22 +5,5 @@
     {
       "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
+        "<arguments_raw, not valid JSON>": "{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\""
       },
       "name": "create_event"

Strategies: nonstream, token, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, nested-json, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event
```json
{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event\n```json\n{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 10028, 76740, 201, 9854, 14479, 201, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-numeric-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v3-numeric-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['calculate'], got []
expected_matchspecial failtool_calls: expected ['calculate'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['calculate'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['calculate'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['calculate'] | rand:1:8 (fail): tool_calls: one [], got ['calculate'] | rand:2:8 (fail): tool_calls: one [], got ['calculate'] | rand:3:8 (fail): tool_calls: one [], got ['calculate'] | rand:4:8 (fail): tool_calls: one [], got ['calculate'] | rand:5:8 (fail): tool_calls: one [], got ['calculate']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-numeric-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,21 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
-      "name": "calculate"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, numeric-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate
```json
{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate\n```json\n{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 93801, 201, 9854, 14479, 201, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v3"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/v3-parallel-two-calls
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search'], got []
expected_matchspecial failtool_calls: expected ['get_weather', 'search'], got []
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n'
expected_matchrand:1:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c'; tool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "'
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search'], got []
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n'
stream_equals_nonstreamrand:1:8 failtool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c'; tool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "'
split_invariance* failvs one: token (fail): content (whitespace only): one None, got '\n'; tool_calls: one [], got ['get_weather', 'search'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather', 'search']
arguments_jsonrand:1:8 fail[1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3': Expecting ',' delimiter: line 1 column 68 (char 67))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "c': Unterminated string starting at: line 1 column 28 (char 27)); [1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "': Unterminated string starting at: line 1 column 60 (char 59))
arguments_schemarand:1:8 fail[1] search: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated; [1] search: arguments are not a JSON object; not validated
parallel_orderone failexpected 2 calls ['get_weather', 'search'], got 0 []
parallel_orderspecial failexpected 2 calls ['get_weather', 'search'], got 0 []

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-parallel-two-calls --strategy one --strategy rand:1:8 --strategy rand:3:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,25 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8

--- expected
+++ observed
@@ -12,12 +12,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3"
       },
       "name": "search"

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"c"
       },
       "name": "get_weather"
@@ -12,12 +11,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \""
       },
       "name": "search"

Strategies: nonstream, rand:2:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Zürich", "unit": "c"}
```<｜tool▁call▁end｜>
<｜tool▁call▁begin｜>function<｜tool▁sep｜>search
```json
{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Zürich\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search\n```json\n{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 201, 128808, 8701, 128814, 14346, 201, 9854, 14479, 201, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 30316, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Paris", "unit": "c"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Paris\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v3-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls: expected ['get_weather'], got []
expected_matchspecial failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls: expected ['get_weather'], got []
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "London"}', got ''
expected_matchrand:2:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
expected_matchrand:3:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
expected_matchrand:5:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamone failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "London"}', got ''
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
split_invariance* failvs one: token (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): content: one None, got 'Let me check the current conditions in'; tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls: one [], got ['get_weather']
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream, token, rand:4:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, text-before-call, x-deepseek-v3.

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "London"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"London\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 27571, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v3-truncated-mid-arguments
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/v3.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Zür"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: truncated, x-deepseek-v3.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Zür", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v3"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v3-unicode-emoji expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v3-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['translate'], got []
expected_matchspecial failtool_calls: expected ['translate'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['translate'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['translate'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['translate'] | rand:1:8 (fail): tool_calls: one [], got ['translate'] | rand:2:8 (fail): tool_calls: one [], got ['translate'] | rand:3:8 (fail): tool_calls: one [], got ['translate'] | rand:4:8 (fail): tool_calls: one [], got ['translate'] | rand:5:8 (fail): tool_calls: one [], got ['translate']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-unicode-emoji --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
-      "name": "translate"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate
```json
{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate\n```json\n{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 77296, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-empty-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-empty-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_time'], got []
expected_matchspecial failtool_calls: expected ['get_time'], got []
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{}', got ''
expected_matchrand:3:8 failtool_calls: expected ['get_time'], got []
expected_matchrand:4:8 failtool_calls: expected ['get_time'], got []
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{}', got ''
stream_equals_nonstreamone failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got ''
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream ['get_time'], got []
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{}', got ''
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_time'] | rand:1:8 (fail): tool_calls: one [], got ['get_time'] | rand:2:8 (fail): tool_calls: one [], got ['get_time'] | rand:5:8 (fail): tool_calls: one [], got ['get_time']
arguments_jsonrand:1:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] get_time: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemarand:1:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_time: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_time: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -4,5 +4,7 @@
   "tool_calls": [
     {
-      "arguments": {},
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
       "name": "get_time"
     }

Strategies: nonstream, token

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, empty-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 20035, 128814, 25902, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-marker-in-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['echo'], got ['f']
expected_matchspecial failtool_calls: expected ['echo'], got ['f']
expected_matchtoken failtool_calls: expected ['echo'], got ['echo', 'f']
expected_matchrand:1:8 failtool_calls: expected ['echo'], got ['echo', 'f']
expected_matchrand:2:8 failtool_calls: expected ['echo'], got ['echo', 'f']
expected_matchrand:3:8 failtool_calls: expected ['echo'], got ['echo', 'f']
expected_matchrand:4:8 failtool_calls: expected ['echo'], got ['echo', 'f']
expected_matchrand:5:8 failtool_calls: expected ['echo'], got ['echo', 'f']
stream_equals_nonstreamone failtool_calls: nonstream ['echo'], got ['f']
stream_equals_nonstreamspecial failtool_calls: nonstream ['echo'], got ['f']
stream_equals_nonstreamtoken failtool_calls: nonstream ['echo'], got ['echo', 'f']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream ['echo'], got ['echo', 'f']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream ['echo'], got ['echo', 'f']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream ['echo'], got ['echo', 'f']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream ['echo'], got ['echo', 'f']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream ['echo'], got ['echo', 'f']
split_invariance* failvs one: token (fail): tool_calls: one ['f'], got ['echo', 'f'] | rand:1:8 (fail): tool_calls: one ['f'], got ['echo', 'f'] | rand:2:8 (fail): tool_calls: one ['f'], got ['echo', 'f'] | rand:3:8 (fail): tool_calls: one ['f'], got ['echo', 'f'] | rand:4:8 (fail): tool_calls: one ['f'], got ['echo', 'f'] | rand:5:8 (fail): tool_calls: one ['f'], got ['echo', 'f']
no_leakagetoken failtool_calls[1].arguments contains '<｜tool▁calls▁end｜>'; tool_calls[1].arguments contains '<｜tool▁call▁end｜>'
no_leakagerand:1:8 failtool_calls[1].arguments contains '<｜tool▁calls▁end｜>'; tool_calls[1].arguments contains '<｜tool▁call▁end｜>'
no_leakagerand:2:8 failtool_calls[1].arguments contains '<｜tool▁calls▁end｜>'; tool_calls[1].arguments contains '<｜tool▁call▁end｜>'
no_leakagerand:4:8 failtool_calls[1].arguments contains '<｜tool▁calls▁end｜>'; tool_calls[1].arguments contains '<｜tool▁call▁end｜>'
no_leakagerand:5:8 failtool_calls[1].arguments contains '<｜tool▁calls▁end｜>'; tool_calls[1].arguments contains '<｜tool▁call▁end｜>'
arguments_jsonone fail[0] f: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] f: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like ': Unterminated string starting at: line 1 column 10 (char 9)); [1] f: arguments are not valid JSON text ('{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>': Extra data: line 1 column 4 (char 3))
arguments_jsonrand:1:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like': Unterminated string starting at: line 1 column 10 (char 9)); [1] f: arguments are not valid JSON text ('{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>': Extra data: line 1 column 4 (char 3))
arguments_jsonrand:2:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like ': Unterminated string starting at: line 1 column 10 (char 9)); [1] f: arguments are not valid JSON text ('{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>': Extra data: line 1 column 4 (char 3))
arguments_jsonrand:3:8 fail[0] echo: arguments are not valid JSON text ('{"text":': Expecting value: line 1 column 9 (char 8)); [1] f: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like ': Unterminated string starting at: line 1 column 10 (char 9)); [1] f: arguments are not valid JSON text ('{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>': Extra data: line 1 column 4 (char 3))
arguments_jsonrand:5:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like ': Unterminated string starting at: line 1 column 10 (char 9)); [1] f: arguments are not valid JSON text ('{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>': Extra data: line 1 column 4 (char 3))
arguments_schemaone fail[0] 'f' is not an offered tool
arguments_schemaspecial fail[0] 'f' is not an offered tool
arguments_schematoken fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool
arguments_schemarand:1:8 fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool
arguments_schemarand:2:8 fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool
arguments_schemarand:3:8 fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool
arguments_schemarand:4:8 fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool
arguments_schemarand:5:8 fail[0] echo: arguments are not a JSON object; not validated; [1] 'f' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -5,7 +5,7 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": ""
       },
-      "name": "echo"
+      "name": "f"
     }
   ]

Strategies: token, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,7 +5,13 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like "
       },
       "name": "echo"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>"
+      },
+      "name": "f"
     }
   ]

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,7 +5,13 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like"
       },
       "name": "echo"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": "{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>"
+      },
+      "name": "f"
     }
   ]

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,7 +5,13 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\":"
       },
       "name": "echo"
+    },
+    {
+      "arguments": {
+        "<arguments_raw, not valid JSON>": ""
+      },
+      "name": "f"
     }
   ]

Strategies: nonstream

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, marker-in-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{\"text\": \"Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 18843, 128814, 24313, 2067, 3362, 582, 105119, 1277, 223, 128808, 72, 128814, 25902, 477, 92637, 2148, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v31"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-multi-turn-after-tool-result expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v31-multi-turn-after-tool-result
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['search'], got []
expected_matchspecial failtool_calls: expected ['search'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['search'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['search'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['search'] | rand:1:8 (fail): tool_calls: one [], got ['search'] | rand:2:8 (fail): tool_calls: one [], got ['search'] | rand:3:8 (fail): tool_calls: one [], got ['search'] | rand:4:8 (fail): tool_calls: one [], got ['search'] | rand:5:8 (fail): tool_calls: one [], got ['search']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-multi-turn-after-tool-result --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "café Zürich"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, multi-turn, unicode, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café Zürich"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-multi-turn-after-tool-result", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café Zürich\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 125388, 616, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-nested-json expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['create_event'], got []
expected_matchspecial failtool_calls: expected ['create_event'], got []
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
stream_equals_nonstreamone failtool_calls: nonstream ['create_event'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['create_event'], got []
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
split_invariance* failvs one: token (fail): tool_calls: one [], got ['create_event'] | rand:1:8 (fail): tool_calls: one [], got ['create_event'] | rand:2:8 (fail): tool_calls: one [], got ['create_event'] | rand:3:8 (fail): tool_calls: one [], got ['create_event'] | rand:4:8 (fail): tool_calls: one [], got ['create_event'] | rand:5:8 (fail): tool_calls: one [], got ['create_event']
arguments_jsonrand:2:8 fail[0] create_event: arguments are not valid JSON text ('{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...: Expecting value: line 1 column 193 (char 192))
arguments_jsonrand:4:8 fail[0] create_event: arguments are not valid JSON text ('{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...: Unterminated string starting at: line 1 column 195 (char 194))
arguments_jsonrand:5:8 fail[0] create_event: arguments are not valid JSON text ('{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...: Expecting value: line 1 column 193 (char 192))
arguments_schemarand:2:8 fail[0] create_event: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] create_event: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] create_event: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-nested-json --strategy one --strategy rand:2:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 11 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,28 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
-      "name": "create_event"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:2:8, rand:5:8

--- expected
+++ observed
@@ -5,22 +5,5 @@
     {
       "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
+        "<arguments_raw, not valid JSON>": "{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\":"
       },
       "name": "create_event"

Strategies: rand:4:8

--- expected
+++ observed
@@ -5,22 +5,5 @@
     {
       "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
+        "<arguments_raw, not valid JSON>": "{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\""
       },
       "name": "create_event"

Strategies: nonstream, token, rand:1:8, rand:3:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, nested-json, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 10028, 76740, 128814, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-numeric-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v31-numeric-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['calculate'], got []
expected_matchspecial failtool_calls: expected ['calculate'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['calculate'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['calculate'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['calculate'] | rand:1:8 (fail): tool_calls: one [], got ['calculate'] | rand:2:8 (fail): tool_calls: one [], got ['calculate'] | rand:3:8 (fail): tool_calls: one [], got ['calculate'] | rand:4:8 (fail): tool_calls: one [], got ['calculate'] | rand:5:8 (fail): tool_calls: one [], got ['calculate']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-numeric-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,21 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
-      "name": "calculate"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, numeric-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 93801, 128814, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v31"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/v31-parallel-two-calls
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search'], got []
expected_matchspecial failtool_calls: expected ['get_weather', 'search'], got []
expected_matchrand:1:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c'; tool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": '
expected_matchrand:3:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "'; tool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max":'
expected_matchrand:5:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"],'
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c'; tool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": '
stream_equals_nonstreamrand:3:8 failtool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "'; tool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max":'
stream_equals_nonstreamrand:5:8 failtool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"],'
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather', 'search'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather', 'search']
arguments_jsonrand:1:8 fail[1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "': Unterminated string starting at: line 1 column 60 (char 59))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "c': Unterminated string starting at: line 1 column 28 (char 27)); [1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": ': Expecting value: line 1 column 67 (char 66))
arguments_jsonrand:3:8 fail[1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3': Expecting ',' delimiter: line 1 column 68 (char 67))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "': Unterminated string starting at: line 1 column 28 (char 27)); [1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max":': Expecting value: line 1 column 66 (char 65))
arguments_jsonrand:5:8 fail[1] search: arguments are not valid JSON text ('{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"],': Expecting property name enclosed in double quotes: line 1 column 59 (char 58))
arguments_schemarand:1:8 fail[1] search: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated; [1] search: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[1] search: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated; [1] search: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[1] search: arguments are not a JSON object; not validated
parallel_orderone failexpected 2 calls ['get_weather', 'search'], got 0 []
parallel_orderspecial failexpected 2 calls ['get_weather', 'search'], got 0 []

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-parallel-two-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,25 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8

--- expected
+++ observed
@@ -12,12 +12,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \""
       },
       "name": "search"

Strategies: rand:2:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"c"
       },
       "name": "get_weather"
@@ -12,12 +11,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": "
       },
       "name": "search"

Strategies: rand:3:8

--- expected
+++ observed
@@ -12,12 +12,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3"
       },
       "name": "search"

Strategies: rand:4:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \""
       },
       "name": "get_weather"
@@ -12,12 +11,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\":"
       },
       "name": "search"

Strategies: rand:5:8

--- expected
+++ observed
@@ -12,12 +12,5 @@
     {
       "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
+        "<arguments_raw, not valid JSON>": "{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"],"
       },
       "name": "search"

Strategies: nonstream, token

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Zürich\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 3831, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v31-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Paris", "unit": "c"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Paris\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls: expected ['get_weather'], got []
expected_matchspecial failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls: expected ['get_weather'], got []
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "London"}', got ''
expected_matchrand:2:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
expected_matchrand:3:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"city": "London"}', got ''
expected_matchrand:5:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'; tool_calls[0].arguments: expected '{"city": "London"}', got ''
stream_equals_nonstreamone failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "London"}', got ''
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"city": "London"}', got ''
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'; tool_calls[0].arguments: nonstream '{"city": "London"}', got ''
split_invariance* failvs one: token (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): content: one None, got 'Let me check the current conditions in'; tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls: one [], got ['get_weather']
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "London"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:4:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "city": "London"
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "get_weather"

Strategies: nonstream, token

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, text-before-call, x-deepseek-v31.

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "London"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"London\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 27571, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-truncated-mid-arguments
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:1:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:2:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:3:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:4:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür' | rand:5:8 (fail): tool_calls[0].arguments: one '', got '{"city": "Zür'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/v31.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Zür"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: truncated, x-deepseek-v31.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Zür", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v31"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v31-unicode-emoji expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v31-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['translate'], got []
expected_matchspecial failtool_calls: expected ['translate'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['translate'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['translate'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['translate'] | rand:1:8 (fail): tool_calls: one [], got ['translate'] | rand:2:8 (fail): tool_calls: one [], got ['translate'] | rand:3:8 (fail): tool_calls: one [], got ['translate'] | rand:4:8 (fail): tool_calls: one [], got ['translate'] | rand:5:8 (fail): tool_calls: one [], got ['translate']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-unicode-emoji --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
-      "name": "translate"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 77296, 128814, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v32-truncated-mid-arguments expected_error, stream_equals_nonstream, arguments_json, arguments_schema
Checks that did not pass for deepseek/v32-truncated-mid-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamtoken failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemanonstream fail[0] get_weather: $: 'city' is a required property
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/v32.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, x-deepseek-v32.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v32"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "model_rule": "DeepSeek-V3\\.2",
  "model_type": "deepseek_v32",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "\": [\"text\"]}}\n</functions>\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "1bade67606d54952ae652a4f71f44065041ef0842ad1b5420ed82f8159502bc8",
    "source": "vLLM tokenizer_mode=deepseek_v32 encoder (vllm.tokenizers.deepseek_v32_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v32",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "requested_mode": "auto",
    "resolved_revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v32",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v32-truncated-second-parallel-call expected_match, arguments_schema
Checks that did not pass for deepseek/v32-truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
arguments_schemanonstream fail[1] search: $: 'query' is a required property
arguments_schemaone fail[1] search: $: 'query' is a required property
arguments_schemaspecial fail[1] search: $: 'query' is a required property
arguments_schematoken fail[1] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:2:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:3:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:4:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 16 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v32"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "model_rule": "DeepSeek-V3\\.2",
  "model_type": "deepseek_v32",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": " \"content\"]}}\n</functions>\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "1bade67606d54952ae652a4f71f44065041ef0842ad1b5420ed82f8159502bc8",
    "source": "vLLM tokenizer_mode=deepseek_v32 encoder (vllm.tokenizers.deepseek_v32_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v32",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "requested_mode": "auto",
    "resolved_revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v32",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v4-truncated-mid-arguments expected_error, stream_equals_nonstream, arguments_json, arguments_schema
Checks that did not pass for deepseek/v4-truncated-mid-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamtoken failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemanonstream fail[0] get_weather: $: 'city' is a required property
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/v4.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, x-deepseek-v4.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v4"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "model_rule": "DeepSeek-V4",
  "model_type": "deepseek_v4",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "bfbb51e563b89c7d3581864cd0ea4d975b96e14fed034a222caf9d50c549b99d",
    "source": "vLLM tokenizer_mode=deepseek_v4 encoder (vllm.tokenizers.deepseek_v4_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v4",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "requested_mode": "auto",
    "resolved_revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v4",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v4-truncated-second-parallel-call expected_match, arguments_schema
Checks that did not pass for deepseek/v4-truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
arguments_schemanonstream fail[1] search: $: 'query' is a required property
arguments_schemaone fail[1] search: $: 'query' is a required property
arguments_schemaspecial fail[1] search: $: 'query' is a required property
arguments_schematoken fail[1] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:2:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:3:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:4:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 17 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v4"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "model_rule": "DeepSeek-V4",
  "model_type": "deepseek_v4",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "bfbb51e563b89c7d3581864cd0ea4d975b96e14fed034a222caf9d50c549b99d",
    "source": "vLLM tokenizer_mode=deepseek_v4 encoder (vllm.tokenizers.deepseek_v4_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v4",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "requested_mode": "auto",
    "resolved_revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v4",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v41-truncated-mid-arguments expected_error, stream_equals_nonstream, arguments_json, arguments_schema
Checks that did not pass for deepseek/v41-truncated-mid-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamtoken failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{}', got '{"city": "'
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemanonstream fail[0] get_weather: $: 'city' is a required property
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v41.jsonl --id deepseek/v41-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/v41.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py. Tags: truncated, x-deepseek-v41.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v41-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "502bdaec8a3fd88ebc24c4721a7038fbe42f2063c664638127056107920035c1", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v41"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v41; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/v41-truncated-second-parallel-call expected_match, arguments_schema
Checks that did not pass for deepseek/v41-truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
arguments_schemanonstream fail[1] search: $: 'query' is a required property
arguments_schemaone fail[1] search: $: 'query' is a required property
arguments_schemaspecial fail[1] search: $: 'query' is a required property
arguments_schematoken fail[1] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:2:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:3:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:4:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v41.jsonl --id deepseek/v41-truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 17 of fixtures/deepseek/v41.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v41.

Raw output


<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">Zürich</｜DSML｜ parameter>
<｜DSML｜ parameter name="unit" string="true">c</｜DSML｜ parameter>
</｜DSML｜ invoke>
<｜DSML｜ invoke name="search">
<｜DSML｜ parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v41-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "502bdaec8a3fd88ebc24c4721a7038fbe42f2063c664638127056107920035c1", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">Zürich</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"unit\" string=\"true\">c</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n<｜DSML｜ invoke name=\"search\">\n<｜DSML｜ parameter name=\"query\"", "output_token_ids": [271, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 30, 128825, 34756, 2329, 1281, 14346, 3816, 30, 128825, 10767, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v41"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v41; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-escaped-strings expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v3-escaped-strings
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['send_message'], got []
expected_matchspecial failtool_calls: expected ['send_message'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['send_message'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['send_message'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['send_message'] | rand:1:8 (fail): tool_calls: one [], got ['send_message'] | rand:2:8 (fail): tool_calls: one [], got ['send_message'] | rand:3:8 (fail): tool_calls: one [], got ['send_message'] | rand:4:8 (fail): tool_calls: one [], got ['send_message'] | rand:5:8 (fail): tool_calls: one [], got ['send_message']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-escaped-strings --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 5 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,13 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "newline": "line1\nline2",
-        "path": "C:\\Users\\file",
-        "text": "He said \"hello\""
-      },
-      "name": "send_message"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "newline": "line1\nline2",
        "path": "C:\\Users\\file",
        "text": "He said \"hello\""
      },
      "name": "send_message"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61. Tags: single-call, string-escapes, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message
```json
{"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "newline": "line1\nline2",
        "path": "C:\\Users\\file",
        "text": "He said \"hello\""
      },
      "name": "send_message"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-escaped-strings", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "send_message", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "path": {"type": "string"}, "newline": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message\n```json\n{\"text\": \"He said \\\"hello\\\"\", \"path\": \"C:\\\\Users\\\\file\", \"newline\": \"line1\\nline2\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 35397, 53735, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 3158, 1359, 36952, 33310, 28609, 1760, 582, 9860, 3362, 582, 37, 111396, 24979, 7387, 8104, 1760, 582, 2839, 1836, 3362, 582, 1836, 19, 5809, 1836, 20, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "send_message", "arguments": {"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}}]}, "tags": ["single-call", "string-escapes", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-malformed-missing-brace expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/vllm-v3-malformed-missing-brace
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo"', got '{"'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo"', got '{"city":'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo"', got ''
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo"', got '{"'
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']
arguments_jsonnonstream fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"': Unterminated string starting at: line 1 column 2 (char 1))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city":': Expecting value: line 1 column 9 (char 8))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"': Unterminated string starting at: line 1 column 2 (char 1))
arguments_schemanonstream fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/malformed.jsonl --id deepseek/vllm-v3-malformed-missing-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 1 of fixtures/deepseek/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, token, rand:4:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Tokyo\""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: rand:1:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: rand:2:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\":"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: rand:3:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": ""
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68. Tags: malformed, x-deepseek-v3.

Expected graceful failure: The arguments JSON is missing its closing brace; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo"
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-malformed-missing-brace", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 2701, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "The arguments JSON is missing its closing brace; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "x-deepseek-v3"], "notes": "malformed_input_outputs[0] of the test. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-malformed-missing-call-tokens stream_equals_nonstream
Checks that did not pass for deepseek/vllm-v3-malformed-missing-call-tokens
CheckStrategyResultDetail
stream_equals_nonstreamone failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamspecial failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamtoken failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamrand:1:8 failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamrand:2:8 failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamrand:3:8 failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamrand:4:8 failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'
stream_equals_nonstreamrand:5:8 failcontent: nonstream None, got 'function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"}\n```'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/malformed.jsonl --id deepseek/vllm-v3-malformed-missing-call-tokens --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/deepseek/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"}\n```",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68. Tags: malformed, x-deepseek-v3.

Expected graceful failure: <｜tool▁call▁begin｜>/<｜tool▁call▁end｜> are missing inside the calls section; no well-formed call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<｜tool▁calls▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo"}
```<｜tool▁calls▁end｜>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-malformed-missing-call-tokens", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"}\n```<｜tool▁calls▁end｜>", "output_token_ids": [128806, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 60676, 9854, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "<｜tool▁call▁begin｜>/<｜tool▁call▁end｜> are missing inside the calls section; no well-formed call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "x-deepseek-v3"], "notes": "malformed_input_outputs[1] of the test. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-parallel-no-newline expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/vllm-v3-parallel-no-newline
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search_hotels'], got []
expected_matchspecial failtool_calls: expected ['get_weather', 'search_hotels'], got []
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{"city": "Tokyo", "unit":'
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search_hotels'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search_hotels'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{"city": "Tokyo", "unit":'
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather', 'search_hotels'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather', 'search_hotels'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather', 'search_hotels'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather', 'search_hotels'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather', 'search_hotels'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather', 'search_hotels']
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo", "unit":': Expecting value: line 1 column 26 (char 25))
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
parallel_orderone failexpected 2 calls ['get_weather', 'search_hotels'], got 0 []
parallel_orderspecial failexpected 2 calls ['get_weather', 'search_hotels'], got 0 []

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-parallel-no-newline --strategy one --strategy rand:1:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 3 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,19 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
-      "name": "search_hotels"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,6 +5,5 @@
     {
       "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Tokyo\", \"unit\":"
       },
       "name": "get_weather"

Strategies: nonstream, token, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "check_in": "2025-01-15",
        "location": "Tokyo"
      },
      "name": "search_hotels"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L31. Tags: parallel-calls, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels
```json
{"location": "Tokyo", "check_in": "2025-01-15"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "check_in": "2025-01-15",
        "location": "Tokyo"
      },
      "name": "search_hotels"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-parallel-no-newline", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L31", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}, {"type": "function", "function": {"name": "search_hotels", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "check_in": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels\n```json\n{\"location\": \"Tokyo\", \"check_in\": \"2025-01-15\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128808, 8701, 128814, 14346, 17462, 347, 1713, 201, 9854, 14479, 201, 24313, 33182, 3362, 582, 106239, 1760, 582, 9547, 12743, 3362, 582, 939, 23, 15, 2616, 15, 856, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}, {"name": "search_hotels", "arguments": {"location": "Tokyo", "check_in": "2025-01-15"}}]}, "tags": ["parallel-calls", "x-deepseek-v3"], "notes": "Calls are adjacent here; the official template separates them with '\\n'. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather'], got []
expected_matchspecial failtool_calls: expected ['get_weather'], got []
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather'], got []
split_invariance* failvs one: token (fail): tool_calls: one [], got ['get_weather'] | rand:1:8 (fail): tool_calls: one [], got ['get_weather'] | rand:2:8 (fail): tool_calls: one [], got ['get_weather'] | rand:3:8 (fail): tool_calls: one [], got ['get_weather'] | rand:4:8 (fail): tool_calls: one [], got ['get_weather'] | rand:5:8 (fail): tool_calls: one [], got ['get_weather']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 2 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L27. Tags: single-call, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L27", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v3-various-data-types expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/vllm-v3-various-data-types
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['test_function'], got []
expected_matchspecial failtool_calls: expected ['test_function'], got []
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...
stream_equals_nonstreamone failtool_calls: nonstream ['test_function'], got []
stream_equals_nonstreamspecial failtool_calls: nonstream ['test_function'], got []
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...
split_invariance* failvs one: token (fail): tool_calls: one [], got ['test_function'] | rand:1:8 (fail): tool_calls: one [], got ['test_function'] | rand:2:8 (fail): tool_calls: one [], got ['test_function'] | rand:3:8 (fail): tool_calls: one [], got ['test_function'] | rand:4:8 (fail): tool_calls: one [], got ['test_function'] | rand:5:8 (fail): tool_calls: one [], got ['test_function']
arguments_jsonrand:3:8 fail[0] test_function: arguments are not valid JSON text ('{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...: Expecting value: line 1 column 208 (char 207))
arguments_jsonrand:4:8 fail[0] test_function: arguments are not valid JSON text ('{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...: Unterminated string starting at: line 1 column 193 (char 192))
arguments_schemarand:3:8 fail[0] test_function: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] test_function: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-various-data-types --strategy one --strategy rand:3:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -2,25 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "array_field": [
-          "a",
-          "b",
-          "c"
-        ],
-        "bool_field": true,
-        "empty_array": [],
-        "empty_object": {},
-        "float_field": 3.14,
-        "int_field": 42,
-        "null_field": null,
-        "object_field": {
-          "nested": "value"
-        },
-        "string_field": "hello"
-      },
-      "name": "test_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,19 +5,5 @@
     {
       "arguments": {
-        "array_field": [
-          "a",
-          "b",
-          "c"
-        ],
-        "bool_field": true,
-        "empty_array": [],
-        "empty_object": {},
-        "float_field": 3.14,
-        "int_field": 42,
-        "null_field": null,
-        "object_field": {
-          "nested": "value"
-        },
-        "string_field": "hello"
+        "<arguments_raw, not valid JSON>": "{\"string_field\": \"hello\", \"int_field\": 42, \"float_field\": 3.14, \"bool_field\": true, \"null_field\": null, \"array_field\": [\"a\", \"b\", \"c\"], \"object_field\": {\"nested\": \"value\"}, \"empty_array\": [], \"empty_object\":"
       },
       "name": "test_function"

Strategies: rand:4:8

--- expected
+++ observed
@@ -5,19 +5,5 @@
     {
       "arguments": {
-        "array_field": [
-          "a",
-          "b",
-          "c"
-        ],
-        "bool_field": true,
-        "empty_array": [],
-        "empty_object": {},
-        "float_field": 3.14,
-        "int_field": 42,
-        "null_field": null,
-        "object_field": {
-          "nested": "value"
-        },
-        "string_field": "hello"
+        "<arguments_raw, not valid JSON>": "{\"string_field\": \"hello\", \"int_field\": 42, \"float_field\": 3.14, \"bool_field\": true, \"null_field\": null, \"array_field\": [\"a\", \"b\", \"c\"], \"object_field\": {\"nested\": \"value\"}, \"empty_array\": [], \"empty_object"
       },
       "name": "test_function"

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "array_field": [
          "a",
          "b",
          "c"
        ],
        "bool_field": true,
        "empty_array": [],
        "empty_object": {},
        "float_field": 3.14,
        "int_field": 42,
        "null_field": null,
        "object_field": {
          "nested": "value"
        },
        "string_field": "hello"
      },
      "name": "test_function"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39. Tags: single-call, numeric-arguments, nested-json, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function
```json
{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "array_field": [
          "a",
          "b",
          "c"
        ],
        "bool_field": true,
        "empty_array": [],
        "empty_object": {},
        "float_field": 3.14,
        "int_field": 42,
        "null_field": null,
        "object_field": {
          "nested": "value"
        },
        "string_field": "hello"
      },
      "name": "test_function"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-various-data-types", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "test_function", "parameters": {"type": "object", "properties": {"string_field": {"type": "string"}, "int_field": {"type": "integer"}, "float_field": {"type": "number"}, "bool_field": {"type": "boolean"}, "null_field": {"type": "null"}, "array_field": {"type": "array"}, "object_field": {"type": "object"}, "empty_array": {"type": "array"}, "empty_object": {"type": "object"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function\n```json\n{\"string_field\": \"hello\", \"int_field\": 42, \"float_field\": 3.14, \"bool_field\": true, \"null_field\": null, \"array_field\": [\"a\", \"b\", \"c\"], \"object_field\": {\"nested\": \"value\"}, \"empty_array\": [], \"empty_object\": {}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 7958, 64334, 201, 9854, 14479, 201, 24313, 4463, 55291, 3362, 582, 33310, 1760, 582, 650, 55291, 3362, 223, 3180, 14, 582, 15891, 55291, 3362, 223, 21, 16, 929, 14, 582, 20600, 55291, 3362, 3292, 14, 582, 16042, 55291, 3362, 5100, 14, 582, 7386, 55291, 3362, 20584, 67, 1760, 582, 68, 1760, 582, 69, 45390, 582, 10325, 55291, 3362, 28612, 80, 12904, 3362, 582, 5197, 87928, 582, 15566, 30549, 3362, 66517, 582, 15566, 55774, 3362, 26102, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "test_function", "arguments": {"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}}]}, "tags": ["single-call", "numeric-arguments", "nested-json", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "model_rule": "DeepSeek-V3",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (V3: no reasoning)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜>",
  "reasoning_parser": null,
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "requested_mode": "auto",
    "resolved_revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v3",
  "units_source": "fixture.output_token_ids"
}
fail deepseek/vllm-v31-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/vllm-v31-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'normal text', got None; tool_calls: expected ['foo'], got []
expected_matchspecial failcontent: expected 'normal text', got None; tool_calls: expected ['foo'], got []
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"x": 1}', got ''
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"x": 1}', got ''
expected_matchrand:3:8 failcontent: expected 'normal text', got None; tool_calls[0].arguments: expected '{"x": 1}', got ''
expected_matchrand:4:8 failcontent: expected 'normal text', got None
expected_matchrand:5:8 failcontent: expected 'normal text', got None; tool_calls[0].arguments: expected '{"x": 1}', got ''
stream_equals_nonstreamone failcontent: nonstream 'normal text', got None; tool_calls: nonstream ['foo'], got []
stream_equals_nonstreamspecial failcontent: nonstream 'normal text', got None; tool_calls: nonstream ['foo'], got []
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"x":1}', got ''
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"x":1}', got ''
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'normal text', got None; tool_calls[0].arguments: nonstream '{"x":1}', got ''
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'normal text', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'normal text', got None; tool_calls[0].arguments: nonstream '{"x":1}', got ''
split_invariance* failvs one: token (fail): content: one None, got 'normal text'; tool_calls: one [], got ['foo'] | rand:1:8 (fail): content: one None, got 'normal text'; tool_calls: one [], got ['foo'] | rand:2:8 (fail): content: one None, got 'normal text'; tool_calls: one [], got ['foo'] | rand:3:8 (fail): tool_calls: one [], got ['foo'] | rand:4:8 (fail): tool_calls: one [], got ['foo'] | rand:5:8 (fail): tool_calls: one [], got ['foo']
arguments_jsonrand:1:8 fail[0] foo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] foo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] foo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] foo: arguments are not valid JSON text ('': Expecting value: line 1 column 1 (char 0))
arguments_schemarand:1:8 fail[0] foo: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] foo: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] foo: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] foo: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v31-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 6 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "normal text",
+  "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "x": 1
-      },
-      "name": "foo"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:2:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "x": 1
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "foo"

Strategies: rand:3:8, rand:5:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": "normal text",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "x": 1
+        "<arguments_raw, not valid JSON>": ""
       },
       "name": "foo"

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "normal text",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream, token

Matches the expected parse.

{
  "content": "normal text",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "foo"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25. Tags: single-call, text-before-call, x-deepseek-v31.

Raw output

normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{"x":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "normal text",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "foo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "foo", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}}}}}], "raw_output": "normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{\"x\":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [13644, 3051, 128806, 128808, 40897, 128814, 24313, 90, 3362, 19, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "normal text", "reasoning_content": null, "tool_calls": [{"name": "foo", "arguments": {"x": 1}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "model_rule": "DeepSeek-V3\\.1",
  "model_type": "deepseek_v3",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<｜begin▁of▁sentence｜><｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
    "source": "tokenizer chat template (HF repo)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "requested_mode": "auto",
    "resolved_revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v31",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/sglang-v32-text-before-empty-invoke expected_match
Checks that did not pass for deepseek/sglang-v32-text-before-empty-invoke
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/sglang-v32-text-before-empty-invoke --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 9 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me get the current date for you.",
+  "content": "Let me get the current date for you.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L1802. Tags: single-call, text-before-call, empty-arguments, x-deepseek-v32.

Raw output

Let me get the current date for you.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_date">
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "Let me get the current date for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_date"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/sglang-v32-text-before-empty-invoke", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L1802", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_date", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "Let me get the current date for you.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_date\">\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [5718, 678, 1178, 270, 2573, 5045, 362, 440, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 22993, 3816, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me get the current date for you.", "reasoning_content": null, "tool_calls": [{"name": "get_date", "arguments": {}}]}, "tags": ["single-call", "text-before-call", "empty-arguments", "x-deepseek-v32"], "notes": "An invoke with no parameter lines (the encoder itself renders an empty line between the tags). output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "model_rule": "DeepSeek-V3\\.2",
  "model_type": "deepseek_v32",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "erties\": {}}}\n</functions>\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "1bade67606d54952ae652a4f71f44065041ef0842ad1b5420ed82f8159502bc8",
    "source": "vLLM tokenizer_mode=deepseek_v32 encoder (vllm.tokenizers.deepseek_v32_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v32",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "requested_mode": "auto",
    "resolved_revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v32",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v32-reasoning-parallel-text-before expected_match
Checks that did not pass for deepseek/v32-reasoning-parallel-text-before
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchone soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchspecial soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchtoken soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:1:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:2:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:3:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:4:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:5:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-reasoning-parallel-text-before --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 11 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look both up.",
+  "content": "I'll look both up.\n\n",
   "reasoning_content": "Two independent lookups: weather and search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: parallel-calls, reasoning, reasoning-prefilled, text-before-call, unicode, nested-json, string-escapes, x-deepseek-v32.

Raw output

Two independent lookups: weather and search.</think>I'll look both up.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-reasoning-parallel-text-before", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "Two independent lookups: weather and search.</think>I'll look both up.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [14959, 8181, 1747, 13953, 28, 9670, 305, 5351, 16, 128799, 43, 5922, 1747, 1952, 890, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "I'll look both up.", "reasoning_content": "Two independent lookups: weather and search.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "text-before-call", "unicode", "nested-json", "string-escapes", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜><think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "model_rule": "DeepSeek-V3\\.2",
  "model_type": "deepseek_v32",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "\": [\"query\"]}}\n</functions>\n<｜User｜>Use the tools to answer.<｜Assistant｜><think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "1bade67606d54952ae652a4f71f44065041ef0842ad1b5420ed82f8159502bc8",
    "source": "vLLM tokenizer_mode=deepseek_v32 encoder (vllm.tokenizers.deepseek_v32_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v32",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "requested_mode": "auto",
    "resolved_revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v32",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v32-text-before-call expected_match
Checks that did not pass for deepseek/v32-text-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, text-before-call, x-deepseek-v32.

Raw output

Let me check the current conditions in London.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">London</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">London</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 27571, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "model_rule": "DeepSeek-V3\\.2",
  "model_type": "deepseek_v32",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": ": [\"query\"]}}\n</functions>\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v3",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "1bade67606d54952ae652a4f71f44065041ef0842ad1b5420ed82f8159502bc8",
    "source": "vLLM tokenizer_mode=deepseek_v32 encoder (vllm.tokenizers.deepseek_v32_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v32",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "requested_mode": "auto",
    "resolved_revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v32",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v4-reasoning-parallel-text-before expected_match
Checks that did not pass for deepseek/v4-reasoning-parallel-text-before
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchone soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchspecial soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchtoken soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:1:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:2:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:3:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:4:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:5:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-reasoning-parallel-text-before --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 12 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look both up.",
+  "content": "I'll look both up.\n\n",
   "reasoning_content": "Two independent lookups: weather and search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: parallel-calls, reasoning, reasoning-prefilled, text-before-call, unicode, nested-json, string-escapes, x-deepseek-v4.

Raw output

Two independent lookups: weather and search.</think>I'll look both up.

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-reasoning-parallel-text-before", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "Two independent lookups: weather and search.</think>I'll look both up.\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [14959, 8181, 1747, 13953, 28, 9670, 305, 5351, 16, 128822, 43, 5922, 1747, 1952, 890, 339, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "I'll look both up.", "reasoning_content": "Two independent lookups: weather and search.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "text-before-call", "unicode", "nested-json", "string-escapes", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜><think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "model_rule": "DeepSeek-V4",
  "model_type": "deepseek_v4",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "hemas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜><think>",
  "reasoning_parser": "deepseek_v4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "bfbb51e563b89c7d3581864cd0ea4d975b96e14fed034a222caf9d50c549b99d",
    "source": "vLLM tokenizer_mode=deepseek_v4 encoder (vllm.tokenizers.deepseek_v4_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v4",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "requested_mode": "auto",
    "resolved_revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v4",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v4-text-before-call expected_match
Checks that did not pass for deepseek/v4-text-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, text-before-call, x-deepseek-v4.

Raw output

Let me check the current conditions in London.

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">London</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">London</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 339, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 27571, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "model_rule": "DeepSeek-V4",
  "model_type": "deepseek_v4",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v4",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "bfbb51e563b89c7d3581864cd0ea4d975b96e14fed034a222caf9d50c549b99d",
    "source": "vLLM tokenizer_mode=deepseek_v4 encoder (vllm.tokenizers.deepseek_v4_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v4",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "requested_mode": "auto",
    "resolved_revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v4",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v41-reasoning-parallel-text-before expected_match
Checks that did not pass for deepseek/v41-reasoning-parallel-text-before
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchone soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchspecial soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchtoken soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:1:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:2:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:3:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:4:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"
expected_matchrand:5:8 soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n"

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v41.jsonl --id deepseek/v41-reasoning-parallel-text-before --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 12 of fixtures/deepseek/v41.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look both up.",
+  "content": "I'll look both up.\n\n",
   "reasoning_content": "Two independent lookups: weather and search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py. Tags: parallel-calls, reasoning, reasoning-prefilled, text-before-call, unicode, nested-json, string-escapes, x-deepseek-v41.

Raw output

Two independent lookups: weather and search.</think>I'll look both up.

<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">Zürich</｜DSML｜ parameter>
<｜DSML｜ parameter name="unit" string="true">c</｜DSML｜ parameter>
</｜DSML｜ invoke>
<｜DSML｜ invoke name="search">
<｜DSML｜ parameter name="query" string="true">café "best"</｜DSML｜ parameter>
<｜DSML｜ parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜ parameter>
</｜DSML｜ invoke>
</｜DSML｜ calls>

Expected parse

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v41-reasoning-parallel-text-before", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "502bdaec8a3fd88ebc24c4721a7038fbe42f2063c664638127056107920035c1", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "Two independent lookups: weather and search.</think>I'll look both up.\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">Zürich</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"unit\" string=\"true\">c</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n<｜DSML｜ invoke name=\"search\">\n<｜DSML｜ parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n</｜DSML｜ calls>", "output_token_ids": [14959, 8181, 1747, 13953, 28, 9670, 305, 5351, 16, 128822, 43, 5922, 1747, 1952, 890, 339, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 30, 128825, 34756, 2329, 1281, 14346, 3816, 30, 128825, 10767, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 1718, 128825, 10699, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "I'll look both up.", "reasoning_content": "Two independent lookups: weather and search.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "text-before-call", "unicode", "nested-json", "string-escapes", "x-deepseek-v41"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v41; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜><think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "hemas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜><think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/v41-text-before-call expected_match
Checks that did not pass for deepseek/v41-text-before-call
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/v41.jsonl --id deepseek/v41-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 4 of fixtures/deepseek/v41.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py. Tags: single-call, text-before-call, x-deepseek-v41.

Raw output

Let me check the current conditions in London.

<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">London</｜DSML｜ parameter>
</｜DSML｜ invoke>
</｜DSML｜ calls>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v41-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/dba1be0a40aa45a94ad051997016db3960a90277/encoding/encoding.py", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "502bdaec8a3fd88ebc24c4721a7038fbe42f2063c664638127056107920035c1", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">London</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n</｜DSML｜ calls>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 339, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 27571, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 1718, 128825, 10699, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v41"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v41; history render of the official encoder."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/vllm-v41-reasoning-text-before-parallel-calls expected_match
Checks that did not pass for deepseek/vllm-v41-reasoning-text-before-parallel-calls
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v41-reasoning-text-before-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 8 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Checking.",
+  "content": "Checking.\n\n",
   "reasoning_content": "Plan.",
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/engine/test_deepseek_v41.py#L59. Tags: parallel-calls, text-before-call, reasoning, reasoning-prefilled, unicode, numeric-arguments, x-deepseek-v41.

Raw output

Plan.</think>Checking.

<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">杭州</｜DSML｜ parameter>
<｜DSML｜ parameter name="count" string="false">42</｜DSML｜ parameter>
</｜DSML｜ invoke>
<｜DSML｜ invoke name="add">
<｜DSML｜ parameter name="x" string="false">1.5</｜DSML｜ parameter>
<｜DSML｜ parameter name="y" string="false">2.25</｜DSML｜ parameter>
</｜DSML｜ invoke>
</｜DSML｜ calls>

Expected parse

{
  "content": "Checking.",
  "reasoning_content": "Plan.",
  "tool_calls": [
    {
      "arguments": {
        "city": "杭州",
        "count": 42
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "x": 1.5,
        "y": 2.25
      },
      "name": "add"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v41-reasoning-text-before-parallel-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/engine/test_deepseek_v41.py#L59", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}, {"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}}}}], "raw_output": "Plan.</think>Checking.\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">杭州</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"count\" string=\"false\">42</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n<｜DSML｜ invoke name=\"add\">\n<｜DSML｜ parameter name=\"x\" string=\"false\">1.5</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"y\" string=\"false\">2.25</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n</｜DSML｜ calls>", "output_token_ids": [31002, 16, 128822, 123452, 339, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 24463, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 10499, 4, 3418, 1281, 19836, 3320, 3180, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 30, 128825, 34756, 2329, 1281, 3327, 3816, 30, 128825, 10767, 2329, 1281, 90, 4, 3418, 1281, 19836, 3320, 19, 16, 23, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 91, 4, 3418, 1281, 19836, 3320, 20, 16, 1069, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 1718, 128825, 10699, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "Checking.", "reasoning_content": "Plan.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "杭州", "count": 42}}, {"name": "add", "arguments": {"x": 1.5, "y": 2.25}}]}, "tags": ["parallel-calls", "text-before-call", "reasoning", "reasoning-prefilled", "unicode", "numeric-arguments", "x-deepseek-v41"], "notes": "text = 'Plan.</think>' + 'Checking.' + CALLS (CALLS defined at line 18), thinking=True. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "thinking": true
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜><think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "hemas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜><think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}
soft pass deepseek/vllm-v41-text-before-parallel-calls expected_match
Checks that did not pass for deepseek/vllm-v41-text-before-parallel-calls
CheckStrategyResultDetail
expected_matchnonstream soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchone soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchspecial soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchtoken soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'
expected_matchrand:5:8 soft passcontent (whitespace only): expected 'Checking.', got 'Checking.\n\n'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v41-text-before-parallel-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 7 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Checking.",
+  "content": "Checking.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/engine/test_deepseek_v41.py#L59. Tags: parallel-calls, text-before-call, unicode, numeric-arguments, x-deepseek-v41.

Raw output

Checking.

<｜DSML｜ calls>
<｜DSML｜ invoke name="get_weather">
<｜DSML｜ parameter name="city" string="true">杭州</｜DSML｜ parameter>
<｜DSML｜ parameter name="count" string="false">42</｜DSML｜ parameter>
</｜DSML｜ invoke>
<｜DSML｜ invoke name="add">
<｜DSML｜ parameter name="x" string="false">1.5</｜DSML｜ parameter>
<｜DSML｜ parameter name="y" string="false">2.25</｜DSML｜ parameter>
</｜DSML｜ invoke>
</｜DSML｜ calls>

Expected parse

{
  "content": "Checking.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "杭州",
        "count": 42
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "x": 1.5,
        "y": 2.25
      },
      "name": "add"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v41-text-before-parallel-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4.1-Flash"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/engine/test_deepseek_v41.py#L59", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}, {"type": "function", "function": {"name": "add", "parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}}}}], "raw_output": "Checking.\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"get_weather\">\n<｜DSML｜ parameter name=\"city\" string=\"true\">杭州</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"count\" string=\"false\">42</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n<｜DSML｜ invoke name=\"add\">\n<｜DSML｜ parameter name=\"x\" string=\"false\">1.5</｜DSML｜ parameter>\n<｜DSML｜ parameter name=\"y\" string=\"false\">2.25</｜DSML｜ parameter>\n</｜DSML｜ invoke>\n</｜DSML｜ calls>", "output_token_ids": [123452, 339, 30, 128825, 10699, 1018, 30, 128825, 34756, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 10767, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 24463, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 10499, 4, 3418, 1281, 19836, 3320, 3180, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 30, 128825, 34756, 2329, 1281, 3327, 3816, 30, 128825, 10767, 2329, 1281, 90, 4, 3418, 1281, 19836, 3320, 19, 16, 23, 1718, 128825, 10767, 1018, 30, 128825, 10767, 2329, 1281, 91, 4, 3418, 1281, 19836, 3320, 20, 16, 1069, 1718, 128825, 10767, 1018, 1718, 128825, 34756, 1018, 1718, 128825, 10699, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "revision": "dba1be0a40aa45a94ad051997016db3960a90277", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Checking.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "杭州", "count": 42}}, {"name": "add", "arguments": {"x": 1.5, "y": 2.25}}]}, "tags": ["parallel-calls", "text-before-call", "unicode", "numeric-arguments", "x-deepseek-v41"], "notes": "text = 'Checking.' + CALLS (CALLS defined at line 18), thinking=False. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<｜Assistant｜></think>",
  "generation_prompt_match": true,
  "is_harmony": false,
  "model": "deepseek-ai/DeepSeek-V4.1-Flash",
  "model_rule": "DeepSeek-V4\\.1",
  "model_type": "deepseek_v41",
  "parser_source": "docs/formats/deepseek.md (vLLM v0.30.0 row)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "emas to invoke tool calls.\n<｜User｜>Use the tools to answer.<｜Assistant｜></think>",
  "reasoning_parser": "deepseek_v41",
  "stop_token_in_final_delta": false,
  "template": {
    "sha256": "081382cb953a1c8041fdfc82a6001cf0c932553171436625c89fe5b9fbcda464",
    "source": "vLLM tokenizer_mode=deepseek_v41 encoder (vllm.tokenizers.deepseek_v41_encoding)"
  },
  "tokenizer": {
    "mode": "deepseek_v41",
    "repo": "deepseek-ai/DeepSeek-V4.1-Flash",
    "requested_mode": "auto",
    "resolved_revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "revision": "dba1be0a40aa45a94ad051997016db3960a90277",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "deepseek_v41",
  "units_source": "fixture.output_token_ids"
}