CanIToolCall caniuse.com for tool calling

DeepSeek (V3/R1, V3.1, V3.2 DSML, V4 DSML, V4.1 DSML) on sglang 0.5.20

fail 20% strict pass 13 pass · 23 soft pass · 28 fail · 21 unsupported

Run

Engine
sglang 0.5.20
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}
Results file
data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 11 23 24 0 19%
expected_error 2 0 4 0 33%
stream_equals_nonstream 13 25 26 0 20%
split_invariance 13 29 22 0 20%
no_leakage 60 0 4 0 94%
arguments_json 47 0 6 0 89%
arguments_schema 33 0 20 0 62%
parallel_order 4 0 3 0 57%

Fixtures needing attention

fail deepseek/v3-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-marker-in-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "Reply as ```json\\n{\\"a\\": 1}\\n``` or use <｜tool▁sep｜> tokens."}'
arguments_schemaone fail[0] echo: $: 'text' is a required property
arguments_schemaspecial fail[0] echo: $: 'text' is a required property
arguments_schemarand:2:8 fail[0] echo: $: 'text' is a required property
arguments_schemarand:3:8 fail[0] echo: $: 'text' is a required property
arguments_schemarand:4:8 fail[0] echo: $: 'text' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-marker-in-arguments --strategy one --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:2:8, rand:3:8, rand:4:8

--- expected
+++ observed
@@ -4,7 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
-      },
+      "arguments": {},
       "name": "echo"
     }

Strategies: nonstream, token, rand:1:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
      },
      "name": "echo"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, marker-in-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo
```json
{"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>echo\n```json\n{\"text\": \"Reply as ```json\\n{\\\"a\\\": 1}\\n``` or use <｜tool▁sep｜> tokens.\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 18843, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 19905, 412, 35739, 14479, 5809, 3757, 4, 67, 55550, 223, 19, 1464, 80, 9854, 469, 1347, 223, 128814, 35977, 2148, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Reply as ```json\n{\"a\": 1}\n``` or use <｜tool▁sep｜> tokens."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v3"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v3-nested-json expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
arguments_schemaone fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property
arguments_schemaspecial fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property
arguments_schemarand:2:8 fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property
arguments_schemarand:4:8 fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-nested-json --strategy one --strategy rand:2:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:2:8, rand:4:8

--- expected
+++ observed
@@ -4,24 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
+      "arguments": {},
       "name": "create_event"
     }

Strategies: nonstream, token, rand:1:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, nested-json, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event
```json
{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>create_event\n```json\n{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 10028, 76740, 201, 9854, 14479, 201, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v3-numeric-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-numeric-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}'
arguments_schemaone fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemaspecial fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemarand:1:8 fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemarand:2:8 fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemarand:4:8 fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemarand:5:8 fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-numeric-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -4,17 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
+      "arguments": {},
       "name": "calculate"
     }

Strategies: nonstream, token, rand:3:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, numeric-arguments, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate
```json
{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>calculate\n```json\n{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 93801, 201, 9854, 14479, 201, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v3"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v3-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/v3-parallel-two-calls
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search'], got ['search']
expected_matchspecial failtool_calls: expected ['get_weather', 'search'], got ['search']
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n'
expected_matchrand:1:8 failtool_calls: expected ['get_weather', 'search'], got ['get_weather']
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{"query": "café \\"best\\"", "filters": {"tags": ["a", "...; tool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{}'
expected_matchrand:5:8 failtool_calls[1].arguments: expected '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{}'
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search'], got ['search']
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search'], got ['search']
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n'
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"city": "Zürich", "unit": "c"}', got '{"city": "Zürich", "unit": "c"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{"query": "café \\"best\\"", "filters": {"tags": ["a", "...; tool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{}'
stream_equals_nonstreamrand:5:8 failtool_calls[1].arguments: nonstream '{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "max": 3}}', got '{}'
split_invariance* failvs one: token (fail): content (whitespace only): one None, got '\n'; tool_calls: one ['search'], got ['get_weather', 'search'] | rand:1:8 (fail): tool_calls: one ['search'], got ['get_weather'] | rand:2:8 (fail): tool_calls: one ['search'], got ['get_weather', 'search'] | rand:3:8 (fail): tool_calls: one ['search'], got ['get_weather', 'search'] | rand:4:8 (fail): tool_calls: one ['search'], got ['get_weather', 'search'] | rand:5:8 (fail): tool_calls: one ['search'], got ['get_weather', 'search']
no_leakagerand:1:8 failtool_calls[0].arguments contains '<｜tool▁call▁begin｜>'; tool_calls[0].arguments contains '<｜tool▁call▁end｜>'; tool_calls[0].arguments contains '<｜tool▁sep｜>'
no_leakagerand:3:8 failtool_calls[0].arguments contains '<｜tool▁call▁begin｜>'; tool_calls[0].arguments contains '<｜tool▁call▁end｜>'; tool_calls[0].arguments contains '<｜tool▁sep｜>'
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "c"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{"query": "café \\"best\\"", "filters": {"tags": ["a", "...: Extra data: line 2 column 1 (char 32))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "c"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{"query": "café \\"best\\"", "filters": {"tags": ["a", "...: Extra data: line 2 column 1 (char 32))
arguments_schemaone fail[0] search: $: 'query' is a required property
arguments_schemaspecial fail[0] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated; [1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property
parallel_orderone failexpected 2 calls ['get_weather', 'search'], got 1 ['search']
parallel_orderspecial failexpected 2 calls ['get_weather', 'search'], got 1 ['search']
parallel_orderrand:1:8 failexpected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-parallel-two-calls --strategy one --strategy rand:1:8 --strategy rand:3:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,21 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
+      "arguments": {},
       "name": "search"
     }

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,21 +5,7 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,20 +5,10 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}"
       },
       "name": "get_weather"
     },
     {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
+      "arguments": {},
       "name": "search"
     }

Strategies: rand:5:8

--- expected
+++ observed
@@ -11,14 +11,5 @@
     },
     {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
+      "arguments": {},
       "name": "search"
     }

Strategies: nonstream, rand:2:8, rand:4:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Zürich", "unit": "c"}
```<｜tool▁call▁end｜>
<｜tool▁call▁begin｜>function<｜tool▁sep｜>search
```json
{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Zürich\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜>\n<｜tool▁call▁begin｜>function<｜tool▁sep｜>search\n```json\n{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 201, 128808, 8701, 128814, 14346, 201, 9854, 14479, 201, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 30316, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v3-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}'
arguments_schemaone fail[0] get_weather: $: 'city' is a required property
arguments_schemaspecial fail[0] get_weather: $: 'city' is a required property
arguments_schemarand:1:8 fail[0] get_weather: $: 'city' is a required property
arguments_schemarand:4:8 fail[0] get_weather: $: 'city' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-single-call --strategy one --strategy rand:1:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:4:8

--- expected
+++ observed
@@ -4,8 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: nonstream, token, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Paris", "unit": "c"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Paris\", \"unit\": \"c\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v3-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchspecial failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchrand:1:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
expected_matchrand:2:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
expected_matchrand:3:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'; tool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchrand:5:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamone failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamspecial failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'; tool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
split_invariance* failvs one: token (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:1:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:3:8 (fail): content: one None, got 'Let me check the current conditions in' | rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.' | rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}'
arguments_schemaone fail[0] get_weather: $: 'city' is a required property
arguments_schemaspecial fail[0] get_weather: $: 'city' is a required property
arguments_schemarand:3:8 fail[0] get_weather: $: 'city' is a required property
arguments_schemarand:4:8 fail[0] get_weather: $: 'city' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,8 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {
-        "city": "London"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,10 +1,8 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {
-        "city": "London"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: rand:4:8

--- expected
+++ observed
@@ -4,7 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "London"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: nonstream, token

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, text-before-call, x-deepseek-v3.

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "London"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"London\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 27571, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v3-unicode-emoji expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v3-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}'
arguments_schemaone fail[0] translate: $: 'text' is a required property
arguments_schemaspecial fail[0] translate: $: 'text' is a required property
arguments_schemarand:1:8 fail[0] translate: $: 'text' is a required property
arguments_schemarand:3:8 fail[0] translate: $: 'text' is a required property
arguments_schemarand:4:8 fail[0] translate: $: 'text' is a required property
arguments_schemarand:5:8 fail[0] translate: $: 'text' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v3.jsonl --id deepseek/v3-unicode-emoji --strategy one --strategy rand:1:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/deepseek/v3.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -4,8 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
+      "arguments": {},
       "name": "translate"
     }

Strategies: nonstream, token, rand:2:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate
```json
{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v3-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/e9b33add76883f293d6bf61f6bd89b497e80e335/tokenizer_config.json", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>translate\n```json\n{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 77296, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v3"], "notes": "Sub-format deepseek-v3; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-marker-in-arguments
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e...; tool_calls: expected ['echo'], got []
expected_matchone failtool_calls: expected ['echo'], got ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f']
expected_matchspecial failtool_calls: expected ['echo'], got ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f']
expected_matchtoken failcontent: expected None, got ' are markup."}'; tool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like <｜tool▁call▁begin｜>f{}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like <｜tool▁call▁begin｜>{} are markup."}{} are markup."}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like{} are markup."}{} are markup."}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like <｜tool▁call▁begin｜>f{} are markup."}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like{} are markup."{} are markup."}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}', got '{"text": "Tokens like <｜tool▁call▁begin｜>{} are markup."}{} are markup."}'
stream_equals_nonstreamone failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f']
stream_equals_nonstreamspecial failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f']
stream_equals_nonstreamtoken failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got ' are markup."}'; tool_calls: nonstream [], got ['echo']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁e..., got None; tool_calls: nonstream [], got ['echo']
split_invariance* failvs one: token (fail): content: one None, got ' are markup."}'; tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo'] | rand:1:8 (fail): tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo'] | rand:2:8 (fail): tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo'] | rand:3:8 (fail): tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo'] | rand:4:8 (fail): tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo'] | rand:5:8 (fail): tool_calls: one ['echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f'], got ['echo']
no_leakagenonstream failcontent contains '<｜tool▁calls▁begin｜>'; content contains '<｜tool▁calls▁end｜>'; content contains '<｜tool▁call▁begin｜>'; content contains '<｜tool▁call▁end｜>'; content contains '<｜tool▁sep｜>'
no_leakageone failtool_calls[0].name contains '<｜tool▁call▁begin｜>'; tool_calls[0].name contains '<｜tool▁sep｜>'
no_leakagespecial failtool_calls[0].name contains '<｜tool▁call▁begin｜>'; tool_calls[0].name contains '<｜tool▁sep｜>'
arguments_jsontoken fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like <｜tool▁call▁begin｜>f{}': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like <｜tool▁call▁begin｜>{} are markup."}{} are markup."}': Extra data: line 1 column 58 (char 57))
arguments_jsonrand:2:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like{} are markup."}{} are markup."}': Extra data: line 1 column 38 (char 37))
arguments_jsonrand:4:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like{} are markup."{} are markup."}': Expecting ',' delimiter: line 1 column 37 (char 36))
arguments_jsonrand:5:8 fail[0] echo: arguments are not valid JSON text ('{"text": "Tokens like <｜tool▁call▁begin｜>{} are markup."}{} are markup."}': Extra data: line 1 column 58 (char 57))
arguments_schemaone fail[0] 'echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f' is not an offered tool
arguments_schemaspecial fail[0] 'echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f' is not an offered tool
arguments_schematoken fail[0] echo: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] echo: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] echo: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] echo: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] echo: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{\"text\": \"Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
-      },
-      "name": "echo"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special

--- expected
+++ observed
@@ -4,8 +4,6 @@
   "tool_calls": [
     {
-      "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
-      },
-      "name": "echo"
+      "arguments": {},
+      "name": "echo<｜tool▁sep｜>{\"text\": \"Tokens like <｜tool▁call▁begin｜>f"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": " are markup.\"}",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like <｜tool▁call▁begin｜>f{}"
       },
       "name": "echo"

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like <｜tool▁call▁begin｜>{} are markup.\"}{} are markup.\"}"
       },
       "name": "echo"

Strategies: rand:2:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like{} are markup.\"}{} are markup.\"}"
       },
       "name": "echo"

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "text": "Tokens like <｜tool▁call▁begin｜>f{} are markup."
       },
       "name": "echo"

Strategies: rand:4:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
+        "<arguments_raw, not valid JSON>": "{\"text\": \"Tokens like{} are markup.\"{} are markup.\"}"
       },
       "name": "echo"

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, marker-in-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>echo<｜tool▁sep｜>{\"text\": \"Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup.\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 18843, 128814, 24313, 2067, 3362, 582, 105119, 1277, 223, 128808, 72, 128814, 25902, 477, 92637, 2148, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Tokens like <｜tool▁call▁begin｜>f<｜tool▁sep｜>{} are markup."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v31"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-multi-turn-after-tool-result expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-multi-turn-after-tool-result
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"query": "café Zürich"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"query": "café Zürich"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"query": "café Zürich"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"query": "café Zürich"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"query": "café Zürich"}'
arguments_schemaone fail[0] search: $: 'query' is a required property
arguments_schemaspecial fail[0] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-multi-turn-after-tool-result --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,7 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "query": "café Zürich"
-      },
+      "arguments": {},
       "name": "search"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, multi-turn, unicode, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café Zürich"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-multi-turn-after-tool-result", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café Zürich\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 125388, 616, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      26606,
      91294,
      11,
      128804,
      128799
    ],
    "source": "chat_template"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v31-nested-json expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-nested-json
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam..., got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam... | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"nam...
arguments_schemaone fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property
arguments_schemaspecial fail[0] create_event: $: 'title' is a required property; $: 'when' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-nested-json --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,24 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "attendees": [
-          {
-            "name": "Ana",
-            "optional": false
-          },
-          {
-            "name": "Bo",
-            "optional": true,
-            "roles": [
-              "notes"
-            ]
-          }
-        ],
-        "title": "Design review",
-        "when": {
-          "end": "2026-10-01T10:00:00Z",
-          "start": "2026-10-01T09:00:00Z"
-        }
-      },
+      "arguments": {},
       "name": "create_event"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, nested-json, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-nested-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>create_event<｜tool▁sep｜>{\"title\": \"Design review\", \"when\": {\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}, \"attendees\": [{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 10028, 76740, 128814, 24313, 10212, 3362, 582, 31151, 3862, 1760, 582, 21123, 3362, 28612, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 87928, 582, 112191, 5993, 3362, 764, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-numeric-arguments expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-numeric-arguments
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}'
arguments_schemaone fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property
arguments_schemaspecial fail[0] calculate: $: 'x' is a required property; $: 'y' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-numeric-arguments --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,17 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "code": "007",
-        "exact": true,
-        "limit": null,
-        "tolerance": 1e-05,
-        "values": [
-          0,
-          -0.5,
-          1e+21
-        ],
-        "x": -3,
-        "y": 2.5
-      },
+      "arguments": {},
       "name": "calculate"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, numeric-arguments, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>calculate<｜tool▁sep｜>{\"x\": -3, \"y\": 2.5, \"tolerance\": 1e-05, \"exact\": true, \"limit\": null, \"values\": [0, -0.5, 1e+21], \"code\": \"007\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 93801, 128814, 24313, 90, 3362, 565, 21, 14, 582, 91, 3362, 223, 20, 16, 23, 14, 582, 86, 44016, 3362, 223, 19, 71, 15, 2642, 14, 582, 1289, 651, 3362, 3292, 14, 582, 39673, 3362, 5100, 14, 582, 15737, 3362, 764, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 3031, 582, 8308, 3362, 582, 17065, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v31"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/v31-parallel-two-calls
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search'], got ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
expected_matchspecial failtool_calls: expected ['get_weather', 'search'], got ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
expected_matchrand:2:8 failtool_calls: expected ['get_weather', 'search'], got ['get_weather']
expected_matchrand:4:8 failtool_calls: expected ['get_weather', 'search'], got ['get_weather']
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search'], got ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search'], got ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream ['get_weather', 'search'], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather', 'search'] | rand:1:8 (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather', 'search'] | rand:2:8 (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather'] | rand:3:8 (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather', 'search'] | rand:4:8 (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather'] | rand:5:8 (fail): tool_calls: one ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search'], got ['get_weather', 'search']
no_leakageone failtool_calls[0].name contains '<｜tool▁call▁begin｜>'; tool_calls[0].name contains '<｜tool▁call▁end｜>'; tool_calls[0].name contains '<｜tool▁sep｜>'
no_leakagespecial failtool_calls[0].name contains '<｜tool▁call▁begin｜>'; tool_calls[0].name contains '<｜tool▁call▁end｜>'; tool_calls[0].name contains '<｜tool▁sep｜>'
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "c{"query": "{"query": "c{"query": "café \\"{"query": "café \\"best\\"", "filters":{"query": "café \\"best\\"", "filters": {"tags...: Expecting ',' delimiter: line 1 column 32 (char 31))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zürich", "unit": "{"query{"query": "c{"query": "café{"query": "café \\"best{"query": "café \\"best\\"{"query": "café \\"best\\"", "filters": {"tag...: Expecting ',' delimiter: line 1 column 31 (char 30))
arguments_schemaone fail[0] 'get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search' is not an offered tool
arguments_schemaspecial fail[0] 'get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search' is not an offered tool
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
parallel_orderone failexpected 2 calls ['get_weather', 'search'], got 1 ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
parallel_orderspecial failexpected 2 calls ['get_weather', 'search'], got 1 ['get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search']
parallel_orderrand:2:8 failexpected 2 calls ['get_weather', 'search'], got 1 ['get_weather']
parallel_orderrand:4:8 failexpected 2 calls ['get_weather', 'search'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-parallel-two-calls --strategy one --strategy rand:2:8 --strategy rand:4:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,22 +4,6 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
+      "arguments": {},
+      "name": "get_weather<｜tool▁sep｜>{\"city\": \"Zürich\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search"
     }
   ]

Strategies: rand:2:8

--- expected
+++ observed
@@ -5,21 +5,7 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"c{\"query\": \"{\"query\": \"c{\"query\": \"café \\\"{\"query\": \"café \\\"best\\\"\", \"filters\":{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\":{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: rand:4:8

--- expected
+++ observed
@@ -5,21 +5,7 @@
     {
       "arguments": {
-        "city": "Zürich",
-        "unit": "c"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Zürich\", \"unit\": \"{\"query{\"query\": \"c{\"query\": \"café{\"query\": \"café \\\"best{\"query\": \"café \\\"best\\\"{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\":{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
     }
   ]

Strategies: nonstream, token, rand:1:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Zürich", "unit": "c"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Zürich\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>search<｜tool▁sep｜>{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 60, 5421, 616, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128808, 14346, 128814, 24313, 12066, 3362, 582, 69, 2797, 619, 36952, 21910, 28609, 1760, 582, 72, 40332, 3362, 28612, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 3831, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v31-single-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Paris", "unit": "c"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Paris", "unit": "c"}'
arguments_schemaone fail[0] get_weather: $: 'city' is a required property
arguments_schemaspecial fail[0] get_weather: $: 'city' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-single-call --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,8 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Paris", "unit": "c"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Paris\", \"unit\": \"c\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 51119, 1760, 582, 15165, 3362, 582, 69, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-text-before-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchspecial failcontent: expected 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: expected '{"city": "London"}', got '{}'
expected_matchrand:1:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
expected_matchrand:2:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
expected_matchrand:3:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
expected_matchrand:5:8 failcontent: expected 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamone failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamspecial failcontent: nonstream 'Let me check the current conditions in London.', got None; tool_calls[0].arguments: nonstream '{"city": "London"}', got '{}'
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in'
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'Let me check the current conditions in London.', got 'Let me check the current'
split_invariance* failvs one: token (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:1:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:2:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:3:8 (fail): content: one None, got 'Let me check the current conditions in'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:4:8 (fail): content: one None, got 'Let me check the current conditions in London.'; tool_calls[0].arguments: one '{}', got '{"city": "London"}' | rand:5:8 (fail): content: one None, got 'Let me check the current'; tool_calls[0].arguments: one '{}', got '{"city": "London"}'
arguments_schemaone fail[0] get_weather: $: 'city' is a required property
arguments_schemaspecial fail[0] get_weather: $: 'city' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,8 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {
-        "city": "London"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: rand:1:8, rand:2:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream, token, rand:4:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, text-before-call, x-deepseek-v31.

Raw output

Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "London"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"London\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 16, 128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 27571, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v31-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/v31-truncated-mid-arguments
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Zür'
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Zür': Unterminated string starting at: line 1 column 10 (char 9))
arguments_schemaone fail[0] get_weather: $: 'city' is a required property
arguments_schemaspecial fail[0] get_weather: $: 'city' is a required property
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/deepseek/v31.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Zür"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: truncated, x-deepseek-v31.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{"city": "Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Zür", "output_token_ids": [128806, 128808, 1133, 65, 50219, 128814, 24313, 37399, 3362, 582, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v31"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v31-unicode-emoji expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v31-unicode-emoji
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "こんにちは 🌸 مرحبا — naïve café 👩\u200d💻", "target": "de"}'
arguments_schemaone fail[0] translate: $: 'text' is a required property
arguments_schemaspecial fail[0] translate: $: 'text' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v31.jsonl --id deepseek/v31-unicode-emoji --strategy one --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/deepseek/v31.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,8 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "target": "de",
-        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
-      },
+      "arguments": {},
       "name": "translate"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json. Tags: single-call, unicode, x-deepseek-v31.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v31-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/c0781d039fb7a1ba2abc4add0bdc293e92d2b8db/tokenizer_config.json", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>translate<｜tool▁sep｜>{\"text\": \"こんにちは 🌸 مرحبا — naïve café 👩‍💻\", \"target\": \"de\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 77296, 128814, 24313, 2067, 3362, 582, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1760, 582, 28292, 3362, 582, 928, 62773, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v31"], "notes": "Sub-format deepseek-v31; history render of the official chat template."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v32-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-truncated-mid-arguments
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather']
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/deepseek/v32.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zü"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:2:8

{
  "content": "\n\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zü"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, x-deepseek-v32.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v32"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v32-truncated-second-parallel-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v32-truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got []
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failcontent (whitespace only): expected None, got '\n\n\n'; tool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failcontent (whitespace only): expected None, got '\n\n'; tool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamtoken failcontent (whitespace only): nonstream None, got '\n\n\n'; tool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:2:8 failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather', 'search']
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'
arguments_schemaone fail[1] search: $: 'query' is a required property
arguments_schemaspecial fail[1] search: $: 'query' is a required property
arguments_schematoken fail[1] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:2:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:3:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:4:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v32"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v32-unescaped-string-value expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-unescaped-string-value
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchone failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchtoken failcontent (whitespace only): expected None, got '\n\n\n'; tool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:2:8 failcontent (whitespace only): expected None, got '\n\n'; tool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-unescaped-string-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Strategies: token

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, string-escapes, x-unescaped-string, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="write_file">
<｜DSML｜parameter name="path" string="true">a<b>.py</｜DSML｜parameter>
<｜DSML｜parameter name="content" string="true">if a < b and c > d:
    print("x=\"1\"")  # tab	here
</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
        "path": "a<b>.py"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-unescaped-string-value", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"write_file\">\n<｜DSML｜parameter name=\"path\" string=\"true\">a<b>.py</｜DSML｜parameter>\n<｜DSML｜parameter name=\"content\" string=\"true\">if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 25837, 19086, 3816, 30, 128793, 41523, 2329, 1281, 9860, 4, 3418, 1281, 11476, 3320, 67, 93180, 43386, 12403, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 9326, 4, 3418, 1281, 11476, 3320, 394, 260, 818, 291, 305, 274, 1955, 283, 1137, 361, 2777, 1698, 90, 52093, 19, 28609, 5925, 223, 1823, 12826, 200, 1036, 201, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "a<b>.py", "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n"}}]}, "tags": ["single-call", "string-escapes", "x-unescaped-string", "x-deepseek-v32"], "notes": "string=\"true\" values are raw: quotes, <, >, backslashes, tabs and newlines appear unescaped. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/v4-truncated-mid-arguments expected_error, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-truncated-mid-arguments
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output cut by max_tokens inside the first call's arguments: no complete call exists.
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather']
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-mid-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/deepseek/v4.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zü"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:2:8

{
  "content": "\n\n",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zü"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, x-deepseek-v4.

Expected graceful failure: Output cut by max_tokens inside the first call's arguments: no complete call exists. (accept: no_tool_calls, content_passthrough, exception).

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zür
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-truncated-mid-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zür", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected_error": {"reason": "Output cut by max_tokens inside the first call's arguments: no complete call exists.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "x-deepseek-v4"], "notes": "Token prefix of the full render, cut right after 'Zür' appears. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v4-truncated-second-parallel-call expected_match, stream_equals_nonstream, split_invariance, arguments_schema
Checks that did not pass for deepseek/v4-truncated-second-parallel-call
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather'], got []
expected_matchone failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchspecial failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchtoken failcontent (whitespace only): expected None, got '\n\n\n'; tool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:1:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:2:8 failcontent (whitespace only): expected None, got '\n\n'; tool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:3:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:4:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
expected_matchrand:5:8 failtool_calls: expected ['get_weather'], got ['get_weather', 'search']
stream_equals_nonstreamone failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamspecial failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamtoken failcontent (whitespace only): nonstream None, got '\n\n\n'; tool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:1:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:2:8 failcontent (whitespace only): nonstream None, got '\n\n'; tool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:4:8 failtool_calls: nonstream [], got ['get_weather', 'search']
stream_equals_nonstreamrand:5:8 failtool_calls: nonstream [], got ['get_weather', 'search']
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'
arguments_schemaone fail[1] search: $: 'query' is a required property
arguments_schemaspecial fail[1] search: $: 'query' is a required property
arguments_schematoken fail[1] search: $: 'query' is a required property
arguments_schemarand:1:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:2:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:3:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:4:8 fail[1] search: $: 'query' is a required property
arguments_schemarand:5:8 fail[1] search: $: 'query' is a required property

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-truncated-second-parallel-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Zürich",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [
@@ -9,4 +9,8 @@
       },
       "name": "get_weather"
+    },
+    {
+      "arguments": {},
+      "name": "search"
     }
   ]

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: truncated, parallel-calls, unicode, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query"

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-truncated-second-parallel-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\"", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}]}, "tags": ["truncated", "parallel-calls", "unicode", "x-deepseek-v4"], "notes": "The first invoke is complete, the second is cut: a correct parser keeps only the first call. Token prefix of the full render, cut right after 'name=\"query\"' appears. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": false,
    "finish_reason": "length",
    "id": null,
    "kept_by_engine": false,
    "rule": "truncated fixture: finish_reason length",
    "token": null
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/v4-unescaped-string-value expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-unescaped-string-value
CheckStrategyResultDetail
expected_matchnonstream failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchone failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchspecial failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchtoken failcontent (whitespace only): expected None, got '\n\n\n'; tool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:2:8 failcontent (whitespace only): expected None, got '\n\n'; tool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there\\n"}', got '{"path": "a<b>.py", "content": "if a < b and c > d:\\n print(\\"x=\\\\\\"1\\\\\\"\\") # tab\\there"}'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-unescaped-string-value --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 15 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -5,5 +5,5 @@
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Strategies: token

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,9 +1,9 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [
     {
       "arguments": {
-        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
+        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there",
         "path": "a<b>.py"
       },

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, string-escapes, x-unescaped-string, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="write_file">
<｜DSML｜parameter name="path" string="true">a<b>.py</｜DSML｜parameter>
<｜DSML｜parameter name="content" string="true">if a < b and c > d:
    print("x=\"1\"")  # tab	here
</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n",
        "path": "a<b>.py"
      },
      "name": "write_file"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-unescaped-string-value", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a text file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"write_file\">\n<｜DSML｜parameter name=\"path\" string=\"true\">a<b>.py</｜DSML｜parameter>\n<｜DSML｜parameter name=\"content\" string=\"true\">if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 25837, 19086, 3816, 30, 128825, 41523, 2329, 1281, 9860, 4, 3418, 1281, 11476, 3320, 67, 93180, 43386, 12403, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 9326, 4, 3418, 1281, 11476, 3320, 394, 260, 818, 291, 305, 274, 1955, 283, 1137, 361, 2777, 1698, 90, 52093, 19, 28609, 5925, 223, 1823, 12826, 200, 1036, 201, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "write_file", "arguments": {"path": "a<b>.py", "content": "if a < b and c > d:\n    print(\"x=\\\"1\\\"\")  # tab\there\n"}}]}, "tags": ["single-call", "string-escapes", "x-unescaped-string", "x-deepseek-v4"], "notes": "string=\"true\" values are raw: quotes, <, >, backslashes, tabs and newlines appear unescaped. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/vllm-v3-escaped-strings expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v3-escaped-strings
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
expected_matchrand:4:8 failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
expected_matchrand:5:8 failtool_calls[0].arguments: expected '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamrand:4:8 failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
stream_equals_nonstreamrand:5:8 failtool_calls[0].arguments: nonstream '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"text": "He said \\"hello\\"", "path": "C:\\\\Users\\\\file", "newline": "line1\\nline2"}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-escaped-strings --strategy one --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -4,9 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "newline": "line1\nline2",
-        "path": "C:\\Users\\file",
-        "text": "He said \"hello\""
-      },
+      "arguments": {},
       "name": "send_message"
     }

Strategies: nonstream, token, rand:1:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "newline": "line1\nline2",
        "path": "C:\\Users\\file",
        "text": "He said \"hello\""
      },
      "name": "send_message"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61. Tags: single-call, string-escapes, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message
```json
{"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "newline": "line1\nline2",
        "path": "C:\\Users\\file",
        "text": "He said \"hello\""
      },
      "name": "send_message"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-escaped-strings", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L61", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "send_message", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "path": {"type": "string"}, "newline": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>send_message\n```json\n{\"text\": \"He said \\\"hello\\\"\", \"path\": \"C:\\\\Users\\\\file\", \"newline\": \"line1\\nline2\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 35397, 53735, 201, 9854, 14479, 201, 24313, 2067, 3362, 582, 3158, 1359, 36952, 33310, 28609, 1760, 582, 9860, 3362, 582, 37, 111396, 24979, 7387, 8104, 1760, 582, 2839, 1836, 3362, 582, 1836, 19, 5809, 1836, 20, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "send_message", "arguments": {"text": "He said \"hello\"", "path": "C:\\Users\\file", "newline": "line1\nline2"}}]}, "tags": ["single-call", "string-escapes", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/vllm-v3-malformed-missing-brace expected_error, stream_equals_nonstream, split_invariance, arguments_json, arguments_schema
Checks that did not pass for deepseek/vllm-v3-malformed-missing-brace
CheckStrategyResultDetail
expected_errorone failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for The arguments JSON is missing its closing brace; no valid call exists.
stream_equals_nonstreamone failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamspecial failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamtoken failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:1:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:2:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:3:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:4:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
stream_equals_nonstreamrand:5:8 failcontent: nonstream '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{"city": "Tokyo"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>', got None; tool_calls: nonstream [], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo"' | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo"' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo"'
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo"': Expecting ',' delimiter: line 1 column 17 (char 16))
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/malformed.jsonl --id deepseek/vllm-v3-malformed-missing-brace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/deepseek/malformed.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:3:8, rand:5:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:2:8, rand:4:8

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Tokyo\""
      },
      "name": "get_weather"
    }
  ]
}

Strategies: nonstream

{
  "content": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68. Tags: malformed, x-deepseek-v3.

Expected graceful failure: The arguments JSON is missing its closing brace; no valid call exists. (accept: no_tool_calls, content_passthrough).

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo"
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-malformed-missing-brace", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L68", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\"\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 2701, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected_error": {"reason": "The arguments JSON is missing its closing brace; no valid call exists.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "x-deepseek-v3"], "notes": "malformed_input_outputs[0] of the test. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/vllm-v3-parallel-no-newline expected_match, stream_equals_nonstream, split_invariance, no_leakage, arguments_json, arguments_schema, parallel_order
Checks that did not pass for deepseek/vllm-v3-parallel-no-newline
CheckStrategyResultDetail
expected_matchone failtool_calls: expected ['get_weather', 'search_hotels'], got ['search_hotels']
expected_matchspecial failtool_calls: expected ['get_weather', 'search_hotels'], got ['search_hotels']
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{"city": "Tokyo", "unit": "celsius"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{"location": "Tokyo", "check_in": "2025-01-15"}'; tool_calls[1].arguments: expected '{"location": "Tokyo", "check_in": "2025-01-15"}', got '{}'
expected_matchrand:2:8 failtool_calls[1].arguments: expected '{"location": "Tokyo", "check_in": "2025-01-15"}', got '{}'
expected_matchrand:3:8 failtool_calls: expected ['get_weather', 'search_hotels'], got ['get_weather']
stream_equals_nonstreamone failtool_calls: nonstream ['get_weather', 'search_hotels'], got ['search_hotels']
stream_equals_nonstreamspecial failtool_calls: nonstream ['get_weather', 'search_hotels'], got ['search_hotels']
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{"city": "Tokyo", "unit": "celsius"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{"location": "Tokyo", "check_in": "2025-01-15"}'; tool_calls[1].arguments: nonstream '{"location": "Tokyo", "check_in": "2025-01-15"}', got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[1].arguments: nonstream '{"location": "Tokyo", "check_in": "2025-01-15"}', got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls: nonstream ['get_weather', 'search_hotels'], got ['get_weather']
split_invariance* failvs one: token (fail): tool_calls: one ['search_hotels'], got ['get_weather', 'search_hotels'] | rand:1:8 (fail): tool_calls: one ['search_hotels'], got ['get_weather', 'search_hotels'] | rand:2:8 (fail): tool_calls: one ['search_hotels'], got ['get_weather', 'search_hotels'] | rand:3:8 (fail): tool_calls: one ['search_hotels'], got ['get_weather'] | rand:4:8 (fail): tool_calls: one ['search_hotels'], got ['get_weather', 'search_hotels'] | rand:5:8 (fail): tool_calls: one ['search_hotels'], got ['get_weather', 'search_hotels']
no_leakagerand:1:8 failtool_calls[0].arguments contains '<｜tool▁call▁begin｜>'; tool_calls[0].arguments contains '<｜tool▁call▁end｜>'; tool_calls[0].arguments contains '<｜tool▁sep｜>'
no_leakagerand:3:8 failtool_calls[0].arguments contains '<｜tool▁call▁begin｜>'; tool_calls[0].arguments contains '<｜tool▁call▁end｜>'; tool_calls[0].arguments contains '<｜tool▁sep｜>'
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo", "unit": "celsius"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{"location": "Tokyo", "check_in": "2025-01-15"}': Extra data: line 2 column 1 (char 37))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Tokyo", "unit": "celsius"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{"location": "Tokyo", "check_in": "2025-01-15"}': Extra data: line 2 column 1 (char 37))
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
parallel_orderone failexpected 2 calls ['get_weather', 'search_hotels'], got 1 ['search_hotels']
parallel_orderspecial failexpected 2 calls ['get_weather', 'search_hotels'], got 1 ['search_hotels']
parallel_orderrand:3:8 failexpected 2 calls ['get_weather', 'search_hotels'], got 1 ['get_weather']

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-parallel-no-newline --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -4,15 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
+      "arguments": {},
       "name": "search_hotels"
     }

Strategies: rand:1:8

--- expected
+++ observed
@@ -5,14 +5,10 @@
     {
       "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{\"location\": \"Tokyo\", \"check_in\": \"2025-01-15\"}"
       },
       "name": "get_weather"
     },
     {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
+      "arguments": {},
       "name": "search_hotels"
     }

Strategies: rand:2:8

--- expected
+++ observed
@@ -11,8 +11,5 @@
     },
     {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
+      "arguments": {},
       "name": "search_hotels"
     }

Strategies: rand:3:8

--- expected
+++ observed
@@ -5,15 +5,7 @@
     {
       "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
+        "<arguments_raw, not valid JSON>": "{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels{\"location\": \"Tokyo\", \"check_in\": \"2025-01-15\"}"
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "check_in": "2025-01-15",
-        "location": "Tokyo"
-      },
-      "name": "search_hotels"
     }
   ]

Strategies: nonstream, token, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "check_in": "2025-01-15",
        "location": "Tokyo"
      },
      "name": "search_hotels"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L31. Tags: parallel-calls, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels
```json
{"location": "Tokyo", "check_in": "2025-01-15"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "check_in": "2025-01-15",
        "location": "Tokyo"
      },
      "name": "search_hotels"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-parallel-no-newline", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L31", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}, {"type": "function", "function": {"name": "search_hotels", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "check_in": {"type": "string"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_hotels\n```json\n{\"location\": \"Tokyo\", \"check_in\": \"2025-01-15\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128808, 8701, 128814, 14346, 17462, 347, 1713, 201, 9854, 14479, 201, 24313, 33182, 3362, 582, 106239, 1760, 582, 9547, 12743, 3362, 582, 939, 23, 15, 2616, 15, 856, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}, {"name": "search_hotels", "arguments": {"location": "Tokyo", "check_in": "2025-01-15"}}]}, "tags": ["parallel-calls", "x-deepseek-v3"], "notes": "Calls are adjacent here; the official template separates them with '\\n'. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
fail deepseek/vllm-v3-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v3-single-call
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{}'
expected_matchrand:1:8 failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{}'
expected_matchrand:2:8 failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"city": "Tokyo", "unit": "celsius"}', got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{}'
stream_equals_nonstreamrand:1:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{}'
stream_equals_nonstreamrand:2:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"city": "Tokyo", "unit": "celsius"}', got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo", "unit": "celsius"}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo", "unit": "celsius"}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"city": "Tokyo", "unit": "celsius"}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:1:8, rand:2:8, rand:3:8

--- expected
+++ observed
@@ -4,8 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "city": "Tokyo",
-        "unit": "celsius"
-      },
+      "arguments": {},
       "name": "get_weather"
     }

Strategies: nonstream, token, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L27. Tags: single-call, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"city": "Tokyo", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Tokyo",
        "unit": "celsius"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L27", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string"}, "count": {"type": "integer"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n{\"city\": \"Tokyo\", \"unit\": \"celsius\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 1133, 65, 50219, 201, 9854, 14479, 201, 24313, 37399, 3362, 582, 106239, 1760, 582, 15165, 3362, 582, 69, 33030, 60676, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Tokyo", "unit": "celsius"}}]}, "tags": ["single-call", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/vllm-v3-various-data-types expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v3-various-data-types
CheckStrategyResultDetail
expected_matchone failtool_calls[0].arguments: expected '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
expected_matchspecial failtool_calls[0].arguments: expected '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
expected_matchrand:3:8 failtool_calls[0].arguments: expected '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
stream_equals_nonstreamone failtool_calls[0].arguments: nonstream '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
stream_equals_nonstreamspecial failtool_calls[0].arguments: nonstream '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
stream_equals_nonstreamrand:3:8 failtool_calls[0].arguments: nonstream '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne..., got '{}'
split_invariance* failvs one: token (fail): tool_calls[0].arguments: one '{}', got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne... | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne... | rand:2:8 (fail): tool_calls[0].arguments: one '{}', got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne... | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne... | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"ne...

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v3-various-data-types --strategy one --strategy rand:3:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, rand:3:8

--- expected
+++ observed
@@ -4,21 +4,5 @@
   "tool_calls": [
     {
-      "arguments": {
-        "array_field": [
-          "a",
-          "b",
-          "c"
-        ],
-        "bool_field": true,
-        "empty_array": [],
-        "empty_object": {},
-        "float_field": 3.14,
-        "int_field": 42,
-        "null_field": null,
-        "object_field": {
-          "nested": "value"
-        },
-        "string_field": "hello"
-      },
+      "arguments": {},
       "name": "test_function"
     }

Strategies: nonstream, token, rand:1:8, rand:2:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "array_field": [
          "a",
          "b",
          "c"
        ],
        "bool_field": true,
        "empty_array": [],
        "empty_object": {},
        "float_field": 3.14,
        "int_field": 42,
        "null_field": null,
        "object_field": {
          "nested": "value"
        },
        "string_field": "hello"
      },
      "name": "test_function"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39. Tags: single-call, numeric-arguments, nested-json, x-deepseek-v3.

Raw output

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function
```json
{"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "array_field": [
          "a",
          "b",
          "c"
        ],
        "bool_field": true,
        "empty_array": [],
        "empty_object": {},
        "float_field": 3.14,
        "int_field": 42,
        "null_field": null,
        "object_field": {
          "nested": "value"
        },
        "string_field": "hello"
      },
      "name": "test_function"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v3-various-data-types", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3-0324"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv3_tool_parser.py#L39", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "test_function", "parameters": {"type": "object", "properties": {"string_field": {"type": "string"}, "int_field": {"type": "integer"}, "float_field": {"type": "number"}, "bool_field": {"type": "boolean"}, "null_field": {"type": "null"}, "array_field": {"type": "array"}, "object_field": {"type": "object"}, "empty_array": {"type": "array"}, "empty_object": {"type": "object"}}}}}], "raw_output": "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>test_function\n```json\n{\"string_field\": \"hello\", \"int_field\": 42, \"float_field\": 3.14, \"bool_field\": true, \"null_field\": null, \"array_field\": [\"a\", \"b\", \"c\"], \"object_field\": {\"nested\": \"value\"}, \"empty_array\": [], \"empty_object\": {}}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [128806, 128808, 8701, 128814, 7958, 64334, 201, 9854, 14479, 201, 24313, 4463, 55291, 3362, 582, 33310, 1760, 582, 650, 55291, 3362, 223, 3180, 14, 582, 15891, 55291, 3362, 223, 21, 16, 929, 14, 582, 20600, 55291, 3362, 3292, 14, 582, 16042, 55291, 3362, 5100, 14, 582, 7386, 55291, 3362, 20584, 67, 1760, 582, 68, 1760, 582, 69, 45390, 582, 10325, 55291, 3362, 28612, 80, 12904, 3362, 582, 5197, 87928, 582, 15566, 30549, 3362, 66517, 582, 15566, 55774, 3362, 26102, 1778, 9854, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3-0324", "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335", "mode": "hf"}, "generation_prompt": "<｜Assistant｜>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "test_function", "arguments": {"string_field": "hello", "int_field": 42, "float_field": 3.14, "bool_field": true, "null_field": null, "array_field": ["a", "b", "c"], "object_field": {"nested": "value"}, "empty_array": [], "empty_object": {}}}]}, "tags": ["single-call", "numeric-arguments", "nested-json", "x-deepseek-v3"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "22e97ba043dacf9df1f071a9de9d2bb6b0757491f16e96e810834b86e1641286",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3-0324",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": null,
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": null,
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3-0324",
    "revision": "e9b33add76883f293d6bf61f6bd89b497e80e335",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV3Detector",
  "tool_call_parser": "deepseekv3",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
fail deepseek/vllm-v31-text-before-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/vllm-v31-text-before-call
CheckStrategyResultDetail
expected_matchone failcontent: expected 'normal text', got None; tool_calls[0].arguments: expected '{"x": 1}', got '{}'
expected_matchspecial failcontent: expected 'normal text', got None; tool_calls[0].arguments: expected '{"x": 1}', got '{}'
expected_matchrand:1:8 failcontent: expected 'normal text', got None
expected_matchrand:2:8 failcontent: expected 'normal text', got 'normal'
expected_matchrand:3:8 failcontent: expected 'normal text', got None
expected_matchrand:4:8 failcontent: expected 'normal text', got None
expected_matchrand:5:8 failcontent: expected 'normal text', got None
stream_equals_nonstreamone failcontent: nonstream 'normal text', got None; tool_calls[0].arguments: nonstream '{"x": 1}', got '{}'
stream_equals_nonstreamspecial failcontent: nonstream 'normal text', got None; tool_calls[0].arguments: nonstream '{"x": 1}', got '{}'
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'normal text', got None
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'normal text', got 'normal'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'normal text', got None
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'normal text', got None
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'normal text', got None
split_invariance* failvs one: token (fail): content: one None, got 'normal text'; tool_calls[0].arguments: one '{}', got '{"x":1}' | rand:1:8 (fail): tool_calls[0].arguments: one '{}', got '{"x":1}' | rand:2:8 (fail): content: one None, got 'normal'; tool_calls[0].arguments: one '{}', got '{"x":1}' | rand:3:8 (fail): tool_calls[0].arguments: one '{}', got '{"x":1}' | rand:4:8 (fail): tool_calls[0].arguments: one '{}', got '{"x":1}' | rand:5:8 (fail): tool_calls[0].arguments: one '{}', got '{"x":1}'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/vllm-v31-text-before-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,8 @@
 {
-  "content": "normal text",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [
     {
-      "arguments": {
-        "x": 1
-      },
+      "arguments": {},
       "name": "foo"
     }

Strategies: rand:1:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "normal text",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "normal text",
+  "content": "normal",
   "reasoning_content": null,
   "tool_calls": [

Strategies: nonstream, token

Matches the expected parse.

{
  "content": "normal text",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "foo"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25. Tags: single-call, text-before-call, x-deepseek-v31.

Raw output

normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{"x":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Expected parse

{
  "content": "normal text",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "x": 1
      },
      "name": "foo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/vllm-v31-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.1-Terminus"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/tool_parsers/test_deepseekv31_tool_parser.py#L25", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "attribution": "Copyright contributors to the vLLM project"}, "tools": [{"type": "function", "function": {"name": "foo", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}}}}}], "raw_output": "normal text<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>foo<｜tool▁sep｜>{\"x\":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>", "output_token_ids": [13644, 3051, 128806, 128808, 40897, 128814, 24313, 90, 3362, 19, 95, 128809, 128807], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.1", "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "normal text", "reasoning_content": null, "tool_calls": [{"name": "foo", "arguments": {"x": 1}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v31"], "notes": "output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": "deepseek-v3",
    "tool_call_parser": "deepseekv31"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": "45690185067ea25c073bff2d3e425a45ee2564d2bbf6cdf7d885e139d460b8e0",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "model_type": "deepseek_v3"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.1",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param='thinking', default_enabled=False, special_case=None, effort_kwarg=None)",
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.1",
    "revision": "c0781d039fb7a1ba2abc4add0bdc293e92d2b8db",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV31Detector",
  "tool_call_parser": "deepseekv31",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/sglang-v32-text-before-empty-invoke expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/sglang-v32-text-before-empty-invoke
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n\n'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n\n' | rand:4:8 (soft_pass): content (whitespace only): one 'Let me get the current date for you.', got 'Let me get the current date for you.\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/sglang-v32-text-before-empty-invoke --strategy rand:4:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me get the current date for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_date"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me get the current date for you.",
+  "content": "Let me get the current date for you.\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me get the current date for you.",
+  "content": "Let me get the current date for you.\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L1802. Tags: single-call, text-before-call, empty-arguments, x-deepseek-v32.

Raw output

Let me get the current date for you.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_date">
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "Let me get the current date for you.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_date"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/sglang-v32-text-before-empty-invoke", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L1802", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "get_date", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "Let me get the current date for you.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_date\">\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [5718, 678, 1178, 270, 2573, 5045, 362, 440, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 22993, 3816, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me get the current date for you.", "reasoning_content": null, "tool_calls": [{"name": "get_date", "arguments": {}}]}, "tags": ["single-call", "text-before-call", "empty-arguments", "x-deepseek-v32"], "notes": "An invoke with no parameter lines (the encoder itself renders an empty line between the tags). output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/sglang-v4-self-closing-invoke expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/sglang-v4-self-closing-invoke
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/engine-tests.jsonl --id deepseek/sglang-v4-self-closing-invoke --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/deepseek/engine-tests.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "submit"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258. Tags: single-call, empty-arguments, x-self-closing-invoke, x-deepseek-v4.

Raw output

<｜DSML｜tool_calls>
<｜DSML｜invoke name="submit"/>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "submit"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/sglang-v4-self-closing-invoke", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/sgl-project/sglang/blob/94602c9c2b7cbdb8efd5c52802dac6a1c180089e/test/registered/unit/function_call/test_function_call_parser.py#L2258", "revision": "94602c9c2b7cbdb8efd5c52802dac6a1c180089e", "license": "Apache-2.0", "attribution": "Copyright SGLang Team"}, "tools": [{"type": "function", "function": {"name": "submit", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"submit\"/>\n</｜DSML｜tool_calls>", "output_token_ids": [30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 35380, 39278, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "submit", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-self-closing-invoke", "x-deepseek-v4"], "notes": "SGLang's test states 'V4 emits <｜DSML｜invoke name=\"x\"/> for zero-arg tools'. The official V4 encoder renders an open/close pair instead, and DeepSeek's reference parser rejects this form; kept because the engine reports the model generating it. output_token_ids: tokenizer.encode(raw_output)."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-empty-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-empty-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-empty-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, empty-arguments, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_time">

</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_time\">\n\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 20035, 32572, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-marker-in-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-marker-in-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Write <｜DSML｜invoke name=\"x\"> to call a tool."
      },
      "name": "echo"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, marker-in-arguments, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="echo">
<｜DSML｜parameter name="text" string="true">Write <｜DSML｜invoke name="x"> to call a tool.</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Write <｜DSML｜invoke name=\"x\"> to call a tool."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"echo\">\n<｜DSML｜parameter name=\"text\" string=\"true\">Write <｜DSML｜invoke name=\"x\"> to call a tool.</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 18843, 3816, 30, 128793, 41523, 2329, 1281, 2067, 4, 3418, 1281, 11476, 3320, 21750, 818, 128793, 40148, 5406, 2329, 1281, 90, 3320, 304, 1926, 260, 4105, 14020, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Write <｜DSML｜invoke name=\"x\"> to call a tool."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v32"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-numeric-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-numeric-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-numeric-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, numeric-arguments, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="calculate">
<｜DSML｜parameter name="x" string="false">-3</｜DSML｜parameter>
<｜DSML｜parameter name="y" string="false">2.5</｜DSML｜parameter>
<｜DSML｜parameter name="tolerance" string="false">1e-05</｜DSML｜parameter>
<｜DSML｜parameter name="exact" string="false">true</｜DSML｜parameter>
<｜DSML｜parameter name="limit" string="false">null</｜DSML｜parameter>
<｜DSML｜parameter name="values" string="false">[0, -0.5, 1e+21]</｜DSML｜parameter>
<｜DSML｜parameter name="code" string="true">007</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"calculate\">\n<｜DSML｜parameter name=\"x\" string=\"false\">-3</｜DSML｜parameter>\n<｜DSML｜parameter name=\"y\" string=\"false\">2.5</｜DSML｜parameter>\n<｜DSML｜parameter name=\"tolerance\" string=\"false\">1e-05</｜DSML｜parameter>\n<｜DSML｜parameter name=\"exact\" string=\"false\">true</｜DSML｜parameter>\n<｜DSML｜parameter name=\"limit\" string=\"false\">null</｜DSML｜parameter>\n<｜DSML｜parameter name=\"values\" string=\"false\">[0, -0.5, 1e+21]</｜DSML｜parameter>\n<｜DSML｜parameter name=\"code\" string=\"true\">007</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 93801, 3816, 30, 128793, 41523, 2329, 1281, 90, 4, 3418, 1281, 19836, 3320, 15, 21, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 91, 4, 3418, 1281, 19836, 3320, 20, 16, 23, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 86, 44016, 4, 3418, 1281, 19836, 3320, 19, 71, 15, 2642, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 1289, 651, 4, 3418, 1281, 19836, 3320, 11476, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 39673, 4, 3418, 1281, 19836, 3320, 16042, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15737, 4, 3418, 1281, 19836, 3320, 61, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 63, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 8308, 4, 3418, 1281, 11476, 3320, 17065, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v32"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-parallel-two-calls
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected None, got '\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n\n'
stream_equals_nonstreamrand:1:8 soft passcontent (whitespace only): nonstream None, got '\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n\n' | rand:1:8 (soft_pass): content (whitespace only): one None, got '\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-parallel-two-calls --strategy rand:1:8 --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
soft pass deepseek/v32-reasoning-parallel-text-before expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-reasoning-parallel-text-before
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n\n\n"
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream "I'll look both up.", got "I'll look both up.\n\n\n\n"
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one "I'll look both up.", got "I'll look both up.\n\n\n\n"

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-reasoning-parallel-text-before --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look both up.",
+  "content": "I'll look both up.\n\n\n\n",
   "reasoning_content": "Two independent lookups: weather and search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: parallel-calls, reasoning, reasoning-prefilled, text-before-call, unicode, nested-json, string-escapes, x-deepseek-v32.

Raw output

Two independent lookups: weather and search.</think>I'll look both up.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-reasoning-parallel-text-before", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "Two independent lookups: weather and search.</think>I'll look both up.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [14959, 8181, 1747, 13953, 28, 9670, 305, 5351, 16, 128799, 43, 5922, 1747, 1952, 890, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 30, 128793, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128793, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "I'll look both up.", "reasoning_content": "Two independent lookups: weather and search.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "text-before-call", "unicode", "nested-json", "string-escapes", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": true
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128798
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
soft pass deepseek/v32-reasoning-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-reasoning-single-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:4:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-reasoning-single-call --strategy rand:4:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
   "tool_calls": [

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, reasoning, reasoning-prefilled, x-deepseek-v32.

Raw output

The user wants the current weather in Paris. I'll call get_weather.</think>

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Paris</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-reasoning-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "The user wants the current weather in Paris. I'll call get_weather.</think>\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [671, 3967, 10059, 270, 2573, 9670, 295, 11111, 16, 342, 5922, 1926, 1178, 65, 50219, 16, 128799, 271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 51119, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "x-deepseek-v32"], "notes": "Thinking mode: the prompt ends with <think>, so the output holds only </think>. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": true
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128798
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-single-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-single-call --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Paris</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 51119, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-string-false-json expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-string-false-json
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-string-false-json --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 13 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, nested-json, x-string-false-json, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="create_event">
<｜DSML｜parameter name="title" string="true">Design review</｜DSML｜parameter>
<｜DSML｜parameter name="when" string="false">{"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}</｜DSML｜parameter>
<｜DSML｜parameter name="attendees" string="false">[{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-string-false-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"create_event\">\n<｜DSML｜parameter name=\"title\" string=\"true\">Design review</｜DSML｜parameter>\n<｜DSML｜parameter name=\"when\" string=\"false\">{\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}</｜DSML｜parameter>\n<｜DSML｜parameter name=\"attendees\" string=\"false\">[{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 10028, 76740, 3816, 30, 128793, 41523, 2329, 1281, 10212, 4, 3418, 1281, 11476, 3320, 31151, 3862, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 21123, 4, 3418, 1281, 19836, 3320, 24313, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 4, 24568, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 112191, 5993, 4, 3418, 1281, 19836, 3320, 61, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-string-false-json", "x-deepseek-v32"], "notes": "Objects and arrays are JSON inside string=\"false\" parameters. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-text-before-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-text-before-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n' | rand:4:8 (soft_pass): content (whitespace only): one 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-text-before-call --strategy rand:4:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, text-before-call, x-deepseek-v32.

Raw output

Let me check the current conditions in London.

<｜DSML｜function_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">London</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">London</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 339, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128793, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 27571, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v32-unicode-emoji expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v32-unicode-emoji
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v32.jsonl --id deepseek/v32-unicode-emoji --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/deepseek/v32.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py. Tags: single-call, unicode, x-deepseek-v32.

Raw output


<｜DSML｜function_calls>
<｜DSML｜invoke name="translate">
<｜DSML｜parameter name="text" string="true">こんにちは 🌸 مرحبا — naïve café 👩‍💻</｜DSML｜parameter>
<｜DSML｜parameter name="target" string="true">de</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜function_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v32-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V3.2"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/encoding/encoding_dsv32.py", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "5e068c2ba2a6e5ebe37a49bb005650c507e7935d77a32f3f7c11ee071498b370", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "\n\n<｜DSML｜function_calls>\n<｜DSML｜invoke name=\"translate\">\n<｜DSML｜parameter name=\"text\" string=\"true\">こんにちは 🌸 مرحبا — naïve café 👩‍💻</｜DSML｜parameter>\n<｜DSML｜parameter name=\"target\" string=\"true\">de</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜function_calls>", "output_token_ids": [271, 30, 128793, 8701, 4941, 12548, 1018, 30, 128793, 40148, 5406, 2329, 1281, 77296, 3816, 30, 128793, 41523, 2329, 1281, 2067, 4, 3418, 1281, 11476, 3320, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1718, 128793, 41523, 1018, 30, 128793, 41523, 2329, 1281, 28292, 4, 3418, 1281, 11476, 3320, 928, 1718, 128793, 41523, 1018, 1718, 128793, 40148, 5406, 1018, 1718, 128793, 8701, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V3.2", "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v32"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v32; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv32",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV32ForCausalLM"
    ],
    "model_type": "deepseek_v32"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V3.2",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128799
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "_DeepSeekV3Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v3",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "LlamaTokenizer",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V3.2",
    "revision": "a7e62ac04ecb2c0a54d736dc46601c5606cf10a6",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV32Detector",
  "tool_call_parser": "deepseekv32",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-empty-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-empty-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-empty-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, empty-arguments, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_time">

</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-empty-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time. Takes no arguments.", "parameters": {"type": "object", "properties": {}}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_time\">\n\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 20035, 32572, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "empty-arguments", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-marker-in-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-marker-in-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-marker-in-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Write <｜DSML｜invoke name=\"x\"> inside <｜DSML｜tool_calls> to call a tool."
      },
      "name": "echo"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, marker-in-arguments, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="echo">
<｜DSML｜parameter name="text" string="true">Write <｜DSML｜invoke name="x"> inside <｜DSML｜tool_calls> to call a tool.</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "text": "Write <｜DSML｜invoke name=\"x\"> inside <｜DSML｜tool_calls> to call a tool."
      },
      "name": "echo"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-marker-in-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "echo", "description": "Echo a string back verbatim.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"echo\">\n<｜DSML｜parameter name=\"text\" string=\"true\">Write <｜DSML｜invoke name=\"x\"> inside <｜DSML｜tool_calls> to call a tool.</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 18843, 3816, 30, 128825, 41523, 2329, 1281, 2067, 4, 3418, 1281, 11476, 3320, 21750, 818, 128825, 40148, 5406, 2329, 1281, 90, 3320, 6352, 818, 128825, 72461, 4941, 12548, 32, 304, 1926, 260, 4105, 14020, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "echo", "arguments": {"text": "Write <｜DSML｜invoke name=\"x\"> inside <｜DSML｜tool_calls> to call a tool."}}]}, "tags": ["single-call", "marker-in-arguments", "x-deepseek-v4"], "notes": "The argument contains format-marker text, which the official tokenizer maps to special-token ids. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-multi-turn-after-tool-result expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-multi-turn-after-tool-result
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:3:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:3:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:3:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-multi-turn-after-tool-result --strategy rand:3:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "It is 20C. Now the café search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": "It is 20C. Now the café search.",
   "tool_calls": [

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": "It is 20C. Now the café search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, multi-turn, unicode, reasoning, reasoning-prefilled, x-deepseek-v4.

Raw output

It is 20C. Now the café search.</think>

<｜DSML｜tool_calls>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café Zürich</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": "It is 20C. Now the café search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "café Zürich"
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-multi-turn-after-tool-result", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "It is 20C. Now the café search.</think>\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café Zürich</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [2107, 344, 223, 397, 37, 16, 6583, 270, 57664, 5351, 16, 128822, 271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 125388, 616, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": null, "reasoning_content": "It is 20C. Now the café search.", "tool_calls": [{"name": "search", "arguments": {"query": "café Zürich"}}]}, "tags": ["single-call", "multi-turn", "unicode", "reasoning", "reasoning-prefilled", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": true
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128821
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
soft pass deepseek/v4-numeric-arguments expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-numeric-arguments
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-numeric-arguments --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, numeric-arguments, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="calculate">
<｜DSML｜parameter name="x" string="false">-3</｜DSML｜parameter>
<｜DSML｜parameter name="y" string="false">2.5</｜DSML｜parameter>
<｜DSML｜parameter name="tolerance" string="false">1e-05</｜DSML｜parameter>
<｜DSML｜parameter name="exact" string="false">true</｜DSML｜parameter>
<｜DSML｜parameter name="limit" string="false">null</｜DSML｜parameter>
<｜DSML｜parameter name="values" string="false">[0, -0.5, 1e+21]</｜DSML｜parameter>
<｜DSML｜parameter name="code" string="true">007</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "code": "007",
        "exact": true,
        "limit": null,
        "tolerance": 1e-05,
        "values": [
          0,
          -0.5,
          1e+21
        ],
        "x": -3,
        "y": 2.5
      },
      "name": "calculate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-numeric-arguments", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate numbers.", "parameters": {"type": "object", "properties": {"x": {"type": "integer"}, "y": {"type": "number"}, "tolerance": {"type": "number"}, "exact": {"type": "boolean"}, "limit": {"type": ["integer", "null"]}, "values": {"type": "array", "items": {"type": "number"}}, "code": {"type": "string"}}, "required": ["x", "y"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"calculate\">\n<｜DSML｜parameter name=\"x\" string=\"false\">-3</｜DSML｜parameter>\n<｜DSML｜parameter name=\"y\" string=\"false\">2.5</｜DSML｜parameter>\n<｜DSML｜parameter name=\"tolerance\" string=\"false\">1e-05</｜DSML｜parameter>\n<｜DSML｜parameter name=\"exact\" string=\"false\">true</｜DSML｜parameter>\n<｜DSML｜parameter name=\"limit\" string=\"false\">null</｜DSML｜parameter>\n<｜DSML｜parameter name=\"values\" string=\"false\">[0, -0.5, 1e+21]</｜DSML｜parameter>\n<｜DSML｜parameter name=\"code\" string=\"true\">007</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 93801, 3816, 30, 128825, 41523, 2329, 1281, 90, 4, 3418, 1281, 19836, 3320, 15, 21, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 91, 4, 3418, 1281, 19836, 3320, 20, 16, 23, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 86, 44016, 4, 3418, 1281, 19836, 3320, 19, 71, 15, 2642, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 1289, 651, 4, 3418, 1281, 19836, 3320, 11476, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 39673, 4, 3418, 1281, 19836, 3320, 16042, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15737, 4, 3418, 1281, 19836, 3320, 61, 18, 14, 565, 18, 16, 23, 14, 223, 19, 71, 13, 1602, 63, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 8308, 4, 3418, 1281, 11476, 3320, 17065, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "calculate", "arguments": {"x": -3, "y": 2.5, "tolerance": 1e-05, "exact": true, "limit": null, "values": [0, -0.5, 1e+21], "code": "007"}}]}, "tags": ["single-call", "numeric-arguments", "x-deepseek-v4"], "notes": "'code' is the STRING \"007\"; numbers, booleans and null must keep their JSON types. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-parallel-two-calls expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-parallel-two-calls
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n\n'
expected_matchrand:1:8 soft passcontent (whitespace only): expected None, got '\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n\n'
stream_equals_nonstreamrand:1:8 soft passcontent (whitespace only): nonstream None, got '\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n\n' | rand:1:8 (soft_pass): content (whitespace only): one None, got '\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-parallel-two-calls --strategy rand:1:8 --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:1:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: parallel-calls, unicode, nested-json, string-escapes, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-parallel-two-calls", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "unicode", "nested-json", "string-escapes", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
soft pass deepseek/v4-reasoning-parallel-text-before expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-reasoning-parallel-text-before
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected "I'll look both up.", got "I'll look both up.\n\n\n\n"
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream "I'll look both up.", got "I'll look both up.\n\n\n\n"
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one "I'll look both up.", got "I'll look both up.\n\n\n\n"

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-reasoning-parallel-text-before --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "I'll look both up.",
+  "content": "I'll look both up.\n\n\n\n",
   "reasoning_content": "Two independent lookups: weather and search.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: parallel-calls, reasoning, reasoning-prefilled, text-before-call, unicode, nested-json, string-escapes, x-deepseek-v4.

Raw output

Two independent lookups: weather and search.</think>I'll look both up.

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Zürich</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
<｜DSML｜invoke name="search">
<｜DSML｜parameter name="query" string="true">café "best"</｜DSML｜parameter>
<｜DSML｜parameter name="filters" string="false">{"tags": ["a", "b"], "max": 3}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": "I'll look both up.",
  "reasoning_content": "Two independent lookups: weather and search.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Zürich",
        "unit": "c"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-reasoning-parallel-text-before", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}], "raw_output": "Two independent lookups: weather and search.</think>I'll look both up.\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Zürich</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n<｜DSML｜invoke name=\"search\">\n<｜DSML｜parameter name=\"query\" string=\"true\">café \"best\"</｜DSML｜parameter>\n<｜DSML｜parameter name=\"filters\" string=\"false\">{\"tags\": [\"a\", \"b\"], \"max\": 3}</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [14959, 8181, 1747, 13953, 28, 9670, 305, 5351, 16, 128822, 43, 5922, 1747, 1952, 890, 339, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 60, 5421, 616, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 30, 128825, 40148, 5406, 2329, 1281, 14346, 3816, 30, 128825, 41523, 2329, 1281, 12066, 4, 3418, 1281, 11476, 3320, 69, 2797, 619, 582, 21910, 4, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 72, 40332, 4, 3418, 1281, 19836, 3320, 24313, 44175, 3362, 20584, 67, 1760, 582, 68, 45390, 582, 9047, 3362, 223, 21, 24568, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": "I'll look both up.", "reasoning_content": "Two independent lookups: weather and search.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Zürich", "unit": "c"}}, {"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["parallel-calls", "reasoning", "reasoning-prefilled", "text-before-call", "unicode", "nested-json", "string-escapes", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": true
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128821
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}
soft pass deepseek/v4-reasoning-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-reasoning-single-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:4:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-reasoning-single-call --strategy rand:4:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
   "tool_calls": [

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, reasoning, reasoning-prefilled, x-deepseek-v4.

Raw output

The user wants the current weather in Paris. I'll call get_weather.</think>

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Paris</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-reasoning-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "The user wants the current weather in Paris. I'll call get_weather.</think>\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [671, 3967, 10059, 270, 2573, 9670, 295, 11111, 16, 342, 5922, 1926, 1178, 65, 50219, 16, 128822, 271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 51119, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜><think>", "thinking": true, "expected": {"content": null, "reasoning_content": "The user wants the current weather in Paris. I'll call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "reasoning-prefilled", "x-deepseek-v4"], "notes": "Thinking mode: the prompt ends with <think>, so the output holds only </think>. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": true
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128821
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": true,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-single-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-single-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-single-call --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">Paris</｜DSML｜parameter>
<｜DSML｜parameter name="unit" string="true">c</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-single-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n<｜DSML｜parameter name=\"unit\" string=\"true\">c</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 51119, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 15165, 4, 3418, 1281, 11476, 3320, 69, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-string-false-json expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-string-false-json
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-string-false-json --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, nested-json, x-string-false-json, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="create_event">
<｜DSML｜parameter name="title" string="true">Design review</｜DSML｜parameter>
<｜DSML｜parameter name="when" string="false">{"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}</｜DSML｜parameter>
<｜DSML｜parameter name="attendees" string="false">[{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "attendees": [
          {
            "name": "Ana",
            "optional": false
          },
          {
            "name": "Bo",
            "optional": true,
            "roles": [
              "notes"
            ]
          }
        ],
        "title": "Design review",
        "when": {
          "end": "2026-10-01T10:00:00Z",
          "start": "2026-10-01T09:00:00Z"
        }
      },
      "name": "create_event"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-string-false-json", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "create_event", "description": "Create a calendar event.", "parameters": {"type": "object", "properties": {"title": {"type": "string"}, "when": {"type": "object"}, "attendees": {"type": "array", "items": {"type": "object"}}}, "required": ["title", "when"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"create_event\">\n<｜DSML｜parameter name=\"title\" string=\"true\">Design review</｜DSML｜parameter>\n<｜DSML｜parameter name=\"when\" string=\"false\">{\"start\": \"2026-10-01T09:00:00Z\", \"end\": \"2026-10-01T10:00:00Z\"}</｜DSML｜parameter>\n<｜DSML｜parameter name=\"attendees\" string=\"false\">[{\"name\": \"Ana\", \"optional\": false}, {\"name\": \"Bo\", \"optional\": true, \"roles\": [\"notes\"]}]</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 10028, 76740, 3816, 30, 128825, 41523, 2329, 1281, 10212, 4, 3418, 1281, 11476, 3320, 31151, 3862, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 21123, 4, 3418, 1281, 19836, 3320, 24313, 10513, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 3130, 28, 504, 28, 504, 60, 1760, 582, 523, 3362, 582, 939, 24, 15, 553, 15, 2616, 54, 553, 28, 504, 28, 504, 60, 4, 24568, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 112191, 5993, 4, 3418, 1281, 19836, 3320, 61, 24313, 2852, 3362, 582, 114991, 1760, 582, 41401, 3362, 5062, 6989, 28612, 2852, 3362, 582, 18056, 1760, 582, 41401, 3362, 3292, 14, 582, 81345, 3362, 20584, 33955, 13747, 37236, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "create_event", "arguments": {"title": "Design review", "when": {"start": "2026-10-01T09:00:00Z", "end": "2026-10-01T10:00:00Z"}, "attendees": [{"name": "Ana", "optional": false}, {"name": "Bo", "optional": true, "roles": ["notes"]}]}}]}, "tags": ["single-call", "nested-json", "x-string-false-json", "x-deepseek-v4"], "notes": "Objects and arrays are JSON inside string=\"false\" parameters. expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-text-before-call expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-text-before-call
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
expected_matchrand:4:8 soft passcontent (whitespace only): expected 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
stream_equals_nonstreamrand:4:8 soft passcontent (whitespace only): nonstream 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n' | rand:4:8 (soft_pass): content (whitespace only): one 'Let me check the current conditions in London.', got 'Let me check the current conditions in London.\n\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-text-before-call --strategy rand:4:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:2:8, rand:3:8, rand:5:8

Matches the expected parse.

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}

Strategies: token, rand:4:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Let me check the current conditions in London.",
+  "content": "Let me check the current conditions in London.\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, text-before-call, x-deepseek-v4.

Raw output

Let me check the current conditions in London.

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="city" string="true">London</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": "Let me check the current conditions in London.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "London"
      },
      "name": "get_weather"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-text-before-call", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}], "raw_output": "Let me check the current conditions in London.\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"get_weather\">\n<｜DSML｜parameter name=\"city\" string=\"true\">London</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [5718, 678, 4085, 270, 2573, 4132, 295, 6693, 339, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 1133, 65, 50219, 3816, 30, 128825, 41523, 2329, 1281, 37399, 4, 3418, 1281, 11476, 3320, 27571, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": "Let me check the current conditions in London.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "London"}}]}, "tags": ["single-call", "text-before-call", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}
soft pass deepseek/v4-unicode-emoji expected_match, stream_equals_nonstream, split_invariance
Checks that did not pass for deepseek/v4-unicode-emoji
CheckStrategyResultDetail
expected_matchtoken soft passcontent (whitespace only): expected None, got '\n\n\n'
expected_matchrand:2:8 soft passcontent (whitespace only): expected None, got '\n\n'
stream_equals_nonstreamtoken soft passcontent (whitespace only): nonstream None, got '\n\n\n'
stream_equals_nonstreamrand:2:8 soft passcontent (whitespace only): nonstream None, got '\n\n'
split_invariance* soft passvs one: token (soft_pass): content (whitespace only): one None, got '\n\n\n' | rand:2:8 (soft_pass): content (whitespace only): one None, got '\n\n'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/deepseek/v4.jsonl --id deepseek/v4-unicode-emoji --strategy rand:2:8 --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/deepseek/v4.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, rand:1:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}

Strategies: token

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n\n",
   "reasoning_content": null,
   "tool_calls": [

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "\n\n",
   "reasoning_content": null,
   "tool_calls": [

Fixture

Provenance: template_render, https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py. Tags: single-call, unicode, x-deepseek-v4.

Raw output


<｜DSML｜tool_calls>
<｜DSML｜invoke name="translate">
<｜DSML｜parameter name="text" string="true">こんにちは 🌸 مرحبا — naïve café 👩‍💻</｜DSML｜parameter>
<｜DSML｜parameter name="target" string="true">de</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "target": "de",
        "text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻"
      },
      "name": "translate"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "deepseek/v4-unicode-emoji", "family": "deepseek", "models": ["deepseek-ai/DeepSeek-V4-Flash", "deepseek-ai/DeepSeek-V4-Pro"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/60d8d70770c6776ff598c94bb586a859a38244f1/encoding/encoding_dsv4.py", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "license": "MIT", "generator": "scripts/fixtures/deepseek/build.py", "template_sha256": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", "attribution": "Copyright (c) 2023 DeepSeek"}, "tools": [{"type": "function", "function": {"name": "translate", "description": "Translate text.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}, "target": {"type": "string"}}, "required": ["text"]}}}], "raw_output": "\n\n<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"translate\">\n<｜DSML｜parameter name=\"text\" string=\"true\">こんにちは 🌸 مرحبا — naïve café 👩‍💻</｜DSML｜parameter>\n<｜DSML｜parameter name=\"target\" string=\"true\">de</｜DSML｜parameter>\n</｜DSML｜invoke>\n</｜DSML｜tool_calls>", "output_token_ids": [271, 30, 128825, 72461, 4941, 12548, 1018, 30, 128825, 40148, 5406, 2329, 1281, 77296, 3816, 30, 128825, 41523, 2329, 1281, 2067, 4, 3418, 1281, 11476, 3320, 4549, 7245, 2298, 12457, 2841, 73369, 119, 112125, 53067, 2136, 112752, 57664, 52780, 105, 46088, 23903, 122, 1718, 128825, 41523, 1018, 30, 128825, 41523, 2329, 1281, 28292, 4, 3418, 1281, 11476, 3320, 928, 1718, 128825, 41523, 1018, 1718, 128825, 40148, 5406, 1018, 1718, 128825, 72461, 4941, 12548, 32], "tokenizer": {"repo": "deepseek-ai/DeepSeek-V4-Flash", "revision": "60d8d70770c6776ff598c94bb586a859a38244f1", "mode": "hf"}, "generation_prompt": "<｜Assistant｜></think>", "thinking": false, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "translate", "arguments": {"text": "こんにちは 🌸 مرحبا — naïve café 👩‍💻", "target": "de"}}]}, "tags": ["single-call", "unicode", "x-deepseek-v4"], "notes": "expected == DeepSeek's reference parse_message_from_completion_text on this output. Sub-format deepseek-v4; history render of the official encoder."}
Parser configuration 
{
  "auto_detected": {
    "reasoning_parser": null,
    "tool_call_parser": null
  },
  "chat_encoding_spec": "dsv4",
  "chat_template_kwargs": {
    "thinking": false
  },
  "chat_template_sha256": null,
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "DeepseekV4ForCausalLM"
    ],
    "model_type": "deepseek_v4"
  },
  "hf_config_error": null,
  "model": "deepseek-ai/DeepSeek-V4-Flash",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      128804,
      128822
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "DeepSeekV4Detector",
  "reasoning_effort": null,
  "reasoning_enabled": false,
  "reasoning_parser": "deepseek-v4",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 1,
    "kept_by_engine": false,
    "rule": "first stop token of the reference model",
    "token": "<｜end▁of▁sentence｜>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": false,
  "template_reasoning_config": null,
  "thinking": false,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "deepseek-ai/DeepSeek-V4-Flash",
    "revision": "60d8d70770c6776ff598c94bb586a859a38244f1",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "DeepSeekV4Detector",
  "tool_call_parser": "deepseekv4",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

Unsupported fixtures

The adapter declined these fixtures because this engine version has no parser for the family or model.