CanIToolCall caniuse.com for tool calling

gpt-oss (Harmony response format) on vllm 0.30.0

fail 90% strict pass 46 pass · 5 fail

Run

Engine
vllm 0.30.0
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "huggingface-hub": "1.33.0",
  "mistral-common": "1.12.0",
  "openai-harmony": "0.0.8",
  "pinned_version": "0.30.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "many",
  "torch": "2.14.0+cpu",
  "transformers": "5.17.0",
  "wheel": "vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl",
  "wheel_sha256": "ef52ee58c410ead0b8afb190838fa4cbcb52075596f67862a03859d984966ac4"
}
Results file
data/vllm-0.30.0.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 44 0 1 0 98%
expected_error 3 0 3 0 50%
stream_equals_nonstream 50 0 1 0 98%
split_invariance 51 0 0 0 100%
no_leakage 49 0 2 0 96%
arguments_json 35 0 2 0 95%
arguments_schema 35 0 2 0 95%
parallel_order 1 0 0 0 100%

Fixtures needing attention

fail gpt-oss/bug-garbled-channel-commentary-question expected_error
Checks that did not pass for gpt-oss/bug-garbled-channel-commentary-question
CheckStrategyResultDetail
expected_errornonstream failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorone failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorspecial failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errortoken failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorrand:1:8 failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorrand:2:8 failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorrand:3:8 failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorrand:4:8 failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errorrand:5:8 failoutcome exception 'HarmonyError: unexpected tokens remaining in message header: Some("need temperature difference. Also need outside temperature. Get")'; accepted: ['content_passthrough', 'no_tool_calls']

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/bug-garbled-channel-commentary-question --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 24 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "HarmonyError: unexpected tokens remaining in message header: Some(\"need temperature difference. Also need outside temperature. Get\")",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/ggml-org/llama.cpp/issues/27720. Tags: malformed, truncated, no-call, regression, x-malformed-channel.

Expected graceful failure: Malformed channel header ('commentary?commentary?' with no <|message|>) followed by a well-formed analysis message. There is no tool call; the turn must not error. (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>commentary?commentary?We need temperature difference. Also need outside temperature. Get outdoor temperature.<|end|><|start|>assistant<|channel|>analysis<|message|>Need GetLiveContext for Outdoor Temperature.
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/bug-garbled-channel-commentary-question", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ggml-org/llama.cpp/issues/27720", "revision": "issue-27720", "license": "NOASSERTION", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>commentary?commentary?We need temperature difference. Also need outside temperature. Get outdoor temperature.<|end|><|start|>assistant<|channel|>analysis<|message|>Need GetLiveContext for Outdoor Temperature.", "output_token_ids": [200005, 12606, 815, 30, 12606, 815, 30, 2167, 1309, 12088, 9809, 13, 8820, 1309, 7539, 12088, 13, 2942, 16332, 12088, 13, 200007, 200006, 173781, 200005, 35644, 200008, 23483, 2942, 20770, 2522, 395, 40767, 64495, 13], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Malformed channel header ('commentary?commentary?' with no <|message|>) followed by a well-formed analysis message. There is no tool call; the turn must not error.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated", "no-call", "regression", "x-malformed-channel"], "notes": "Quoted from the issue's log line at 179.42 s, which is cut at '<|end|' by the log; this fixture stops before that partial token."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start|>assistant",
  "generation_prompt_match": true,
  "is_harmony": true,
  "model": "openai/gpt-oss-20b",
  "model_rule": "gpt-oss",
  "model_type": "gpt_oss",
  "parser_source": "vLLM gpt-oss recipe; HarmonyParser (is_harmony)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<|end|><|start|>user<|message|>Use the tools to answer.<|end|><|start|>assistant",
  "reasoning_parser": "openai_gptoss",
  "stop_token_in_final_delta": false,
  "template": {
    "openai_harmony": "0.0.8",
    "source": "openai_harmony via vLLM OnlineRenderer._make_request_with_harmony (no Jinja template)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "openai/gpt-oss-20b",
    "requested_mode": "auto",
    "resolved_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "openai",
  "units_source": "fixture.output_token_ids"
}
fail gpt-oss/harmony-truncated-in-arguments expected_error, arguments_json, arguments_schema
Checks that did not pass for gpt-oss/harmony-truncated-in-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorone failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorspecial failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorrand:1:8 failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorrand:2:8 failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorrand:3:8 failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorrand:4:8 failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errorrand:5:8 failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
arguments_jsonnonstream fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonone fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonspecial fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonrand:1:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonrand:2:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonrand:3:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonrand:4:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsonrand:5:8 fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_schemanonstream fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaone fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemaspecial fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:1:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:2:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:3:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:4:8 fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schemarand:5:8 fail[0] get_weather: arguments are not a JSON object; not validated

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-truncated-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 20 of fixtures/gpt-oss/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: truncated, reasoning, x-recipient-in-role.

Expected graceful failure: Output stopped (max_tokens) inside the tool-call JSON; there is no complete call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris", "unit
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/harmony-truncated-in-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start|>assistant",
  "generation_prompt_match": true,
  "is_harmony": true,
  "model": "openai/gpt-oss-20b",
  "model_rule": "gpt-oss",
  "model_type": "gpt_oss",
  "parser_source": "vLLM gpt-oss recipe; HarmonyParser (is_harmony)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<|end|><|start|>user<|message|>Use the tools to answer.<|end|><|start|>assistant",
  "reasoning_parser": "openai_gptoss",
  "stop_token_in_final_delta": false,
  "template": {
    "openai_harmony": "0.0.8",
    "source": "openai_harmony via vLLM OnlineRenderer._make_request_with_harmony (no Jinja template)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "openai/gpt-oss-20b",
    "requested_mode": "auto",
    "resolved_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "openai",
  "units_source": "fixture.output_token_ids"
}
fail gpt-oss/harmony-truncated-in-header expected_match, no_leakage
Checks that did not pass for gpt-oss/harmony-truncated-in-header
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchone failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchspecial failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchtoken failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchrand:1:8 failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchrand:2:8 failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchrand:3:8 failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchrand:4:8 failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
expected_matchrand:5:8 failcontent: expected None, got '<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json'
no_leakagenonstream failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakageone failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagespecial failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagetoken failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagerand:1:8 failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagerand:2:8 failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagerand:3:8 failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagerand:4:8 failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'
no_leakagerand:5:8 failcontent contains '<|start|>'; content contains '<|channel|>'; content contains '<|constrain|>'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-truncated-in-header --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 22 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
   "tool_calls": []

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: truncated, reasoning, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json

Expected parse

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/harmony-truncated-in-header", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": []}, "tags": ["truncated", "reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). Output stopped inside the tool-call header, before <|message|>: the analysis is complete reasoning and there is no call yet."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start|>assistant",
  "generation_prompt_match": true,
  "is_harmony": true,
  "model": "openai/gpt-oss-20b",
  "model_rule": "gpt-oss",
  "model_type": "gpt_oss",
  "parser_source": "vLLM gpt-oss recipe; HarmonyParser (is_harmony)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<|end|><|start|>user<|message|>Use the tools to answer.<|end|><|start|>assistant",
  "reasoning_parser": "openai_gptoss",
  "stop_token_in_final_delta": false,
  "template": {
    "openai_harmony": "0.0.8",
    "source": "openai_harmony via vLLM OnlineRenderer._make_request_with_harmony (no Jinja template)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "openai/gpt-oss-20b",
    "requested_mode": "auto",
    "resolved_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "openai",
  "units_source": "fixture.output_token_ids"
}
fail gpt-oss/llamacpp-stray-commentary-header expected_error, no_leakage, arguments_json, arguments_schema
Checks that did not pass for gpt-oss/llamacpp-stray-commentary-header
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorone failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorspecial failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errortoken failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorrand:1:8 failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorrand:2:8 failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorrand:3:8 failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorrand:4:8 failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
expected_errorrand:5:8 failreturned 1 tool call(s) ['assistant<|channel|>analysis'] for Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.
no_leakagenonstream failtool_calls[0].name contains '<|channel|>'
no_leakageone failtool_calls[0].name contains '<|channel|>'
no_leakagespecial failtool_calls[0].name contains '<|channel|>'
no_leakagetoken failtool_calls[0].name contains '<|channel|>'
no_leakagerand:1:8 failtool_calls[0].name contains '<|channel|>'
no_leakagerand:2:8 failtool_calls[0].name contains '<|channel|>'
no_leakagerand:3:8 failtool_calls[0].name contains '<|channel|>'
no_leakagerand:4:8 failtool_calls[0].name contains '<|channel|>'
no_leakagerand:5:8 failtool_calls[0].name contains '<|channel|>'
arguments_jsonnonstream fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonone fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonspecial fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsontoken fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:1:8 fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:2:8 fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:3:8 fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:4:8 fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_jsonrand:5:8 fail[0] assistant<|channel|>analysis: arguments are not valid JSON text ("I'm\nthinking": Expecting value: line 1 column 1 (char 0))
arguments_schemanonstream fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemaone fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemaspecial fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schematoken fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemarand:1:8 fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemarand:2:8 fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemarand:3:8 fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemarand:4:8 fail[0] 'assistant<|channel|>analysis' is not an offered tool
arguments_schemarand:5:8 fail[0] 'assistant<|channel|>analysis' is not an offered tool

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-stray-commentary-header --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 17 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "Hello, world!\nWhat's up?",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "I'm\nthinking"
      },
      "name": "assistant<|channel|>analysis"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6540-L6546. Tags: no-call, reasoning, malformed, x-malformed-channel.

Expected graceful failure: Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error. (accept: no_tool_calls).

Raw output

<|channel|>commentary to=assistant<|channel|>analysis<|message|>I'm
thinking<|end|><|start|>assistant<|channel|>final<|message|>Hello, world!
What's up?
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/llamacpp-stray-commentary-header", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6540-L6546", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>commentary to=assistant<|channel|>analysis<|message|>I'm\nthinking<|end|><|start|>assistant<|channel|>final<|message|>Hello, world!\nWhat's up?", "output_token_ids": [200005, 12606, 815, 316, 28, 173781, 200005, 35644, 200008, 15390, 198, 149404, 200007, 200006, 173781, 200005, 17196, 200008, 13225, 11, 2375, 4175, 45350, 869, 30], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\\nthinking' and content 'Hello, world!\\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.", "accept": ["no_tool_calls"]}, "tags": ["no-call", "reasoning", "malformed", "x-malformed-channel"], "notes": "Edge case '<|channel|>commentary to=assistant' before reasoning (the stray header llama.cpp #21286 tolerates)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start|>assistant",
  "generation_prompt_match": true,
  "is_harmony": true,
  "model": "openai/gpt-oss-20b",
  "model_rule": "gpt-oss",
  "model_type": "gpt_oss",
  "parser_source": "vLLM gpt-oss recipe; HarmonyParser (is_harmony)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<|end|><|start|>user<|message|>Use the tools to answer.<|end|><|start|>assistant",
  "reasoning_parser": "openai_gptoss",
  "stop_token_in_final_delta": false,
  "template": {
    "openai_harmony": "0.0.8",
    "source": "openai_harmony via vLLM OnlineRenderer._make_request_with_harmony (no Jinja template)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "openai/gpt-oss-20b",
    "requested_mode": "auto",
    "resolved_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "openai",
  "units_source": "fixture.output_token_ids"
}
fail gpt-oss/vllm-malformed-headers stream_equals_nonstream
Checks that did not pass for gpt-oss/vllm-malformed-headers
CheckStrategyResultDetail
stream_equals_nonstreamone failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamspecial failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamtoken failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamrand:1:8 failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamrand:2:8 failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamrand:3:8 failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamrand:4:8 failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'
stream_equals_nonstreamrand:5:8 failcontent: nonstream 'thinking\n<|start|>assistant<|channel|>final {"answer": "hi"}', got 'thinking<|start|>assistant<|channel|>final {"answer": "hi"}'

Minimal repro

uv run canitoolcall run --engine vllm --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-malformed-headers --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/vllm.sh; this run used vllm 0.30.0. The fixture is line 10 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "thinking<|start|>assistant<|channel|>final {\"answer\": \"hi\"}",
  "reasoning_content": "thinking",
  "tool_calls": []
}

Strategies: nonstream

{
  "content": "thinking\n<|start|>assistant<|channel|>final {\"answer\": \"hi\"}",
  "reasoning_content": "thinking",
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L79-L85. Tags: malformed, no-call, reasoning.

Expected graceful failure: The last message header has no <|message|> token (malformed). There is no tool call; a parser must not crash (vLLM recovers the raw text as content). (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>analysis<|message|>thinking<|end|><|start|>assistant<|channel|>commentary<|message|>thinking<|end|><|start|>assistant<|channel|>final {"answer": "hi"}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/vllm-malformed-headers", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L79-L85", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>thinking<|end|><|start|>assistant<|channel|>commentary<|message|>thinking<|end|><|start|>assistant<|channel|>final {\"answer\": \"hi\"}", "output_token_ids": [200005, 35644, 200008, 149404, 200007, 200006, 173781, 200005, 12606, 815, 200008, 149404, 200007, 200006, 173781, 200005, 17196, 10494, 17021, 1243, 392, 3686, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "The last message header has no <|message|> token (malformed). There is no tool call; a parser must not crash (vLLM recovers the raw text as content).", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "no-call", "reasoning"], "notes": "malformed_msgs_str fixture, joined as in test_malformed_msgs_recovers_raw_content. The trailing stop token <|return|> was removed (raw_output ends before the stop token)."}
Parser configuration 
{
  "chat_template_kwargs": {},
  "detokenizer": "vllm.v1.engine.detokenizer.IncrementalDetokenizer.from_new_request",
  "detokenizer_class": "vllm.v1.engine.detokenizer.FastIncrementalDetokenizer",
  "enable_auto_tools": true,
  "engine": "vllm",
  "entrypoint": "vllm.parser.parser_manager.ParserManager.get_parser",
  "generation_prompt": "<|start|>assistant",
  "generation_prompt_match": true,
  "is_harmony": true,
  "model": "openai/gpt-oss-20b",
  "model_rule": "gpt-oss",
  "model_type": "gpt_oss",
  "parser_source": "vLLM gpt-oss recipe; HarmonyParser (is_harmony)",
  "pinned_version": "0.30.0",
  "prompt_messages": [
    {
      "content": "Use the tools to answer.",
      "role": "user"
    }
  ],
  "prompt_tail": "<|end|><|start|>user<|message|>Use the tools to answer.<|end|><|start|>assistant",
  "reasoning_parser": "openai_gptoss",
  "stop_token_in_final_delta": false,
  "template": {
    "openai_harmony": "0.0.8",
    "source": "openai_harmony via vLLM OnlineRenderer._make_request_with_harmony (no Jinja template)"
  },
  "tokenizer": {
    "mode": "hf",
    "repo": "openai/gpt-oss-20b",
    "requested_mode": "auto",
    "resolved_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "tokenizer_class": "transformers.tokenization_utils_tokenizers.TokenizersBackend"
  },
  "tool_choice": "auto (tools and tool_choice omitted when the fixture offers no tools)",
  "tool_parser": "openai",
  "units_source": "fixture.output_token_ids"
}