CanIToolCall caniuse.com for tool calling

gpt-oss (Harmony response format) on llamacpp a25c9865

fail 84% strict pass 43 pass · 8 fail

Run

Engine
llamacpp a25c9865 (a25c9865fe03c954c93fd755b5d79ae86ba99750)
Run
to
Platform
linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0
Strategies
one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8
Fixtures digest
a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run
Engine details
{
  "build_info": "b1-a25c986",
  "detokenizer": "common_token_to_piece(vocab-only GGUF, special = id in preserved_tokens)",
  "harness": "harnesses/llamacpp/replay.cpp",
  "harness_protocol": 1,
  "pinned_version": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "reasoning_format": "deepseek",
  "template_source": "gguf",
  "tokens_per_step": "one"
}
Results file
data/llamacpp-a25c9865.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
CheckPassSoft FailErrorStrict pass rate
expected_match 42 0 3 0 93%
expected_error 1 0 5 0 17%
stream_equals_nonstream 49 0 2 0 96%
no_leakage 51 0 0 0 100%
arguments_json 35 0 1 0 97%
arguments_schema 35 0 1 0 97%
parallel_order 0 0 1 0 0%

8 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail gpt-oss/bug-garbled-channel-commentary-question expected_error
Checks that did not pass for gpt-oss/bug-garbled-channel-commentary-question
CheckStrategyResultDetail
expected_errornonstream failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errortoken failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/bug-garbled-channel-commentary-question --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 24 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/ggml-org/llama.cpp/issues/27720. Tags: malformed, truncated, no-call, regression, x-malformed-channel.

Expected graceful failure: Malformed channel header ('commentary?commentary?' with no <|message|>) followed by a well-formed analysis message. There is no tool call; the turn must not error. (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>commentary?commentary?We need temperature difference. Also need outside temperature. Get outdoor temperature.<|end|><|start|>assistant<|channel|>analysis<|message|>Need GetLiveContext for Outdoor Temperature.
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/bug-garbled-channel-commentary-question", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ggml-org/llama.cpp/issues/27720", "revision": "issue-27720", "license": "NOASSERTION", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>commentary?commentary?We need temperature difference. Also need outside temperature. Get outdoor temperature.<|end|><|start|>assistant<|channel|>analysis<|message|>Need GetLiveContext for Outdoor Temperature.", "output_token_ids": [200005, 12606, 815, 30, 12606, 815, 30, 2167, 1309, 12088, 9809, 13, 8820, 1309, 7539, 12088, 13, 2942, 16332, 12088, 13, 200007, 200006, 173781, 200005, 35644, 200008, 23483, 2942, 20770, 2522, 395, 40767, 64495, 13], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Malformed channel header ('commentary?commentary?' with no <|message|>) followed by a well-formed analysis message. There is no tool call; the turn must not error.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated", "no-call", "regression", "x-malformed-channel"], "notes": "Quoted from the issue's log line at 179.42 s, which is cut at '<|end|' by the log; this fixture stops before that partial token."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "1744a994e40f41c38095f13b492870a25c04c8d3a9d8ceeac5f40ca29c622bc6",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/bug-garbled-channel-question-marks expected_error
Checks that did not pass for gpt-oss/bug-garbled-channel-question-marks
CheckStrategyResultDetail
expected_errornonstream failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errortoken failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/bug-garbled-channel-question-marks --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 23 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/ggml-org/llama.cpp/issues/27720. Tags: malformed, truncated, no-call, regression, x-malformed-channel.

Expected graceful failure: Malformed channel header '<|channel|>??' before a well-formed analysis message. The report's complaint is that the whole turn errors; a parser should skip the garbage and keep going. (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>??<|end|><|start|>assistant<|channel|>analysis<|message|>We must comply with the gating rules. Input: "should I seal the driveway this weekend"
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/bug-garbled-channel-question-marks", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/ggml-org/llama.cpp/issues/27720", "revision": "issue-27720", "license": "NOASSERTION", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>??<|end|><|start|>assistant<|channel|>analysis<|message|>We must comply with the gating rules. Input: \"should I seal the driveway this weekend\"", "output_token_ids": [200005, 6961, 200007, 200006, 173781, 200005, 35644, 200008, 2167, 2804, 40206, 483, 290, 196147, 9607, 13, 10090, 25, 392, 10618, 357, 30803, 290, 61599, 495, 10668, 1], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Malformed channel header '<|channel|>??' before a well-formed analysis message. The report's complaint is that the whole turn errors; a parser should skip the garbage and keep going.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "truncated", "no-call", "regression", "x-malformed-channel"], "notes": "Quoted from the issue's log line '1.06 W common_chat_peg_parse: unparsed peg-native output: <|channel|>??<|end|>...'. The log truncates the output (' ...'), so this fixture ends where the quoted text ends and is also a truncated output."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "1744a994e40f41c38095f13b492870a25c04c8d3a9d8ceeac5f40ca29c622bc6",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/bug-unknown-channel-comment expected_error
Checks that did not pass for gpt-oss/bug-unknown-channel-comment
CheckStrategyResultDetail
expected_errornonstream failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errortoken failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/bug-unknown-channel-comment --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 22 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: bug_report, https://github.com/vllm-project/vllm/issues/58384. Tags: malformed, no-call, regression, x-malformed-channel.

Expected graceful failure: The model emitted a message on the out-of-spec channel 'comment'. The report asks for it to be dropped (with a warning), never to fail the request. (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>comment<|message|>some text<|end|>
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/bug-unknown-channel-comment", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "bug_report", "source_url": "https://github.com/vllm-project/vllm/issues/58384", "revision": "issue-58384", "license": "NOASSERTION", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py"}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>comment<|message|>some text<|end|>", "output_token_ids": [200005, 12606, 200008, 25231, 2201, 200007], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "The model emitted a message on the out-of-spec channel 'comment'. The report asks for it to be dropped (with a warning), never to fail the request.", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "no-call", "regression", "x-malformed-channel"], "notes": "Raw output rendered with openai-harmony from the issue's own reproduction: Message.from_role_and_content(Role.ASSISTANT, 'some text').with_channel('comment'). The trailing <|end|> is kept as rendered."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|call|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "1744a994e40f41c38095f13b492870a25c04c8d3a9d8ceeac5f40ca29c622bc6",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/harmony-truncated-in-arguments expected_error, arguments_json, arguments_schema
Checks that did not pass for gpt-oss/harmony-truncated-in-arguments
CheckStrategyResultDetail
expected_errornonstream failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
expected_errortoken failreturned 1 tool call(s) ['get_weather'] for Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.
arguments_jsonnonstream fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_jsontoken fail[0] get_weather: arguments are not valid JSON text ('{"city": "Paris", "unit': Unterminated string starting at: line 1 column 19 (char 18))
arguments_schemanonstream fail[0] get_weather: arguments are not a JSON object; not validated
arguments_schematoken fail[0] get_weather: arguments are not a JSON object; not validated

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, arguments_json@one fail, arguments_json@special fail, arguments_json@rand:1:8 fail, arguments_json@rand:2:8 fail, arguments_json@rand:3:8 fail, arguments_json@rand:4:8 fail, arguments_json@rand:5:8 fail, arguments_schema@one fail, arguments_schema@special fail, arguments_schema@rand:1:8 fail, arguments_schema@rand:2:8 fail, arguments_schema@rand:3:8 fail, arguments_schema@rand:4:8 fail, arguments_schema@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-truncated-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 20 of fixtures/gpt-oss/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "<arguments_raw, not valid JSON>": "{\"city\": \"Paris\", \"unit"
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: truncated, reasoning, x-recipient-in-role.

Expected graceful failure: Output stopped (max_tokens) inside the tool-call JSON; there is no complete call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris", "unit
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/harmony-truncated-in-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": null,
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "ee5dbf34525a075783c1a38d215c94e4e27228bd9389a306a6bc87c2ef85f372",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/llamacpp-builtin-python-recipient expected_match, stream_equals_nonstream
Checks that did not pass for gpt-oss/llamacpp-builtin-python-recipient
CheckStrategyResultDetail
expected_matchnonstream failparser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
expected_matchtoken failparser raised 'LlamaCppError: The model produced output that does not match the expected peg-native format'
stream_equals_nonstreamtoken failreasoning_content: nonstream None, got 'I will execute python to say hello'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-builtin-python-recipient --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 16 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

--- expected
+++ observed
@@ -1,5 +1,6 @@
 {
   "content": null,
-  "reasoning_content": "I will execute python to say hello",
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
+  "reasoning_content": null,
   "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
   "content": null,
+  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
   "reasoning_content": "I will execute python to say hello",
   "tool_calls": []

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6529-L6536. Tags: no-call, reasoning, x-builtin-recipient, x-recipient-in-channel.

Raw output

<|channel|>analysis<|message|>I will execute python to say hello<|end|><|start|>assistant<|channel|>commentary to=python <|constrain|>code<|message|>print("hello")

Expected parse

{
  "content": null,
  "reasoning_content": "I will execute python to say hello",
  "tool_calls": []
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/llamacpp-builtin-python-recipient", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6529-L6536", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>analysis<|message|>I will execute python to say hello<|end|><|start|>assistant<|channel|>commentary to=python <|constrain|>code<|message|>print(\"hello\")", "output_token_ids": [200005, 35644, 200008, 40, 738, 15792, 22752, 316, 2891, 40617, 200007, 200006, 173781, 200005, 12606, 815, 316, 28, 29010, 220, 200003, 3056, 200008, 1598, 568, 24912, 1405], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "I will execute python to say hello", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-builtin-recipient", "x-recipient-in-channel"], "notes": "Unsolicited call to the built-in python tool, which was not offered: it is not a function tool call, and llama.cpp returns empty content."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|call|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "53278a826a08c3c0291496ff5dd3fb1560dce4e804a079722171783ccb2c40ed",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/vllm-call-then-final expected_match
Checks that did not pass for gpt-oss/vllm-call-then-final
CheckStrategyResultDetail
expected_matchnonstream failcontent: expected 'Done', got None
expected_matchtoken failcontent: expected 'Done', got None

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-call-then-final --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 7 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Done",
+  "content": null,
   "reasoning_content": "Reasoning about query...",
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718. Tags: single-call, reasoning, text-after-call, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{"query": "vllm"}<|call|><|start|>assistant<|channel|>final<|message|>Done

Expected parse

{
  "content": "Done",
  "reasoning_content": "Reasoning about query...",
  "tool_calls": [
    {
      "arguments": {
        "query": "vllm"
      },
      "name": "search"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/vllm-call-then-final", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{\"query\": \"vllm\"}<|call|><|start|>assistant<|channel|>final<|message|>Done", "output_token_ids": [200005, 35644, 200008, 30377, 289, 1078, 5703, 1008, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 200003, 4108, 200008, 10848, 2975, 1243, 392, 85, 680, 76, 18583, 200012, 200006, 173781, 200005, 17196, 200008, 24537], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Done", "reasoning_content": "Reasoning about query...", "tool_calls": [{"name": "search", "arguments": {"query": "vllm"}}]}, "tags": ["single-call", "reasoning", "text-after-call", "x-recipient-in-role"], "notes": "test_cross_channel_with_tool. As in vllm-sequential-calls, text after <|call|> is only seen when the stop token is not applied."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": true,
  "end_token": "<|call|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [
    29
  ],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "6a6a79aaf98298702a303203e49a8ba64d0e90238f82cd1ccd2052b24d020f21",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/vllm-malformed-headers expected_error, stream_equals_nonstream
Checks that did not pass for gpt-oss/vllm-malformed-headers
CheckStrategyResultDetail
expected_errornonstream failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
expected_errortoken failoutcome exception 'LlamaCppError: The model produced output that does not match the expected peg-native format'; accepted: ['content_passthrough', 'no_tool_calls']
stream_equals_nonstreamtoken failcontent: nonstream None, got 'thinking'; reasoning_content: nonstream None, got 'thinking'

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-malformed-headers --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 10 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:5:8

{
  "content": "thinking",
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": "thinking",
  "tool_calls": []
}

Strategies: rand:4:8

{
  "content": null,
  "exception": "LlamaCppError: The model produced output that does not match the expected peg-native format",
  "reasoning_content": "thinking",
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L79-L85. Tags: malformed, no-call, reasoning.

Expected graceful failure: The last message header has no <|message|> token (malformed). There is no tool call; a parser must not crash (vLLM recovers the raw text as content). (accept: no_tool_calls, content_passthrough).

Raw output

<|channel|>analysis<|message|>thinking<|end|><|start|>assistant<|channel|>commentary<|message|>thinking<|end|><|start|>assistant<|channel|>final {"answer": "hi"}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/vllm-malformed-headers", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L79-L85", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>thinking<|end|><|start|>assistant<|channel|>commentary<|message|>thinking<|end|><|start|>assistant<|channel|>final {\"answer\": \"hi\"}", "output_token_ids": [200005, 35644, 200008, 149404, 200007, 200006, 173781, 200005, 12606, 815, 200008, 149404, 200007, 200006, 173781, 200005, 17196, 10494, 17021, 1243, 392, 3686, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "The last message header has no <|message|> token (malformed). There is no tool call; a parser must not crash (vLLM recovers the raw text as content).", "accept": ["no_tool_calls", "content_passthrough"]}, "tags": ["malformed", "no-call", "reasoning"], "notes": "malformed_msgs_str fixture, joined as in test_malformed_msgs_recovers_raw_content. The trailing stop token <|return|> was removed (raw_output ends before the stop token)."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": true,
  "enable_thinking": true,
  "end_token": "<|call|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "1744a994e40f41c38095f13b492870a25c04c8d3a9d8ceeac5f40ca29c622bc6",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}
fail gpt-oss/vllm-sequential-calls expected_match, parallel_order
Checks that did not pass for gpt-oss/vllm-sequential-calls
CheckStrategyResultDetail
expected_matchnonstream failtool_calls: expected ['get_weather', 'get_time'], got ['get_weather']
expected_matchtoken failtool_calls: expected ['get_weather', 'get_time'], got ['get_weather']
parallel_ordernonstream failexpected 2 calls ['get_weather', 'get_time'], got 1 ['get_weather']
parallel_ordertoken failexpected 2 calls ['get_weather', 'get_time'], got 1 ['get_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine llamacpp --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-sequential-calls --strategy token --observed all

Set up the engine first with scripts/engines/llamacpp.sh; this run used llamacpp a25c9865. The fixture is line 6 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -8,10 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "timezone": "UTC"
-      },
-      "name": "get_time"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745. Tags: parallel-calls, reasoning, x-sequential-calls, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{"location": "Paris"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{"timezone": "UTC"}

Expected parse

{
  "content": null,
  "reasoning_content": "Thinking",
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "timezone": "UTC"
      },
      "name": "get_time"
    }
  ]
}
Fixture record (JSONL, ready to vendor into an engine's tests) 
{"id": "gpt-oss/vllm-sequential-calls", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {"type": "object", "properties": {"timezone": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{\"timezone\": \"UTC\"}", "output_token_ids": [200005, 35644, 200008, 133850, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200003, 4108, 200008, 10848, 7693, 1243, 392, 72782, 18583, 200012, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 200003, 4108, 200008, 10848, 106775, 1243, 392, 32674, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Thinking", "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}, {"name": "get_time", "arguments": {"timezone": "UTC"}}]}, "tags": ["parallel-calls", "reasoning", "x-sequential-calls", "x-recipient-in-role"], "notes": "test_tool_index_across_calls: the two deltas concatenated. Harmony has no parallel-call wrapper; consecutive call messages are separated by <|call|>, which is a stop token, so this output only reaches a parser when the stop is not applied (e.g. ignore_eos, vLLM #50690). The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}
Parser configuration 
{
  "additional_stops": [],
  "chat_parser": "common_chat_templates_apply -> common_chat_parse (PEG)",
  "chat_template_kwargs": {},
  "detokenized_matches_raw_output": false,
  "enable_thinking": true,
  "end_token": "<|call|>",
  "engine_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "eog_positions": [
    24
  ],
  "family_handler": "specialized handler 'GPT-OSS'",
  "fixture_generation_prompt": "<|start|>assistant",
  "fixture_tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "format": "peg-native",
  "generation_prompt": "<|start|>assistant",
  "model": "openai/gpt-oss-20b",
  "parallel_tool_calls": false,
  "preserved_tokens": [
    "<|channel|>",
    "<|constrain|>",
    "<|message|>",
    "<|start|>",
    "<|end|>"
  ],
  "reasoning_format": "deepseek",
  "template_alternatives": {
    "llamacpp": {
      "available": false,
      "reason": "llama.cpp models/templates has no copy named after openai/gpt-oss-20b"
    }
  },
  "template_identical_to": "models/templates/openai-gpt-oss-120b.jinja",
  "template_path": "openai--gpt-oss-20b.vocab.gguf: tokenizer.chat_template",
  "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "template_source": "gguf",
  "template_source_reason": null,
  "template_source_requested": "gguf",
  "template_supports_thinking": true,
  "template_variant": "default",
  "thinking_end_tags": [
    "<|end|>"
  ],
  "thinking_start_tag": "<|channel|>analysis<|message|>",
  "tokenizer": "openai/gpt-oss-20b@6cee5e81ee83917806bbde320786a8fb61efebee",
  "tokenizer_mode": "gguf-vocab-only",
  "tools_sha256": "f5984025390fbae00451d8abfb4b1f5635b50ad0e8b2ac49dc34c5da487b673f",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_env": "llama.cpp requirements-convert_hf_to_gguf.txt",
  "vocab_gguf_converter_patches": null,
  "vocab_gguf_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}