gpt-oss (Harmony response format) on ollama 7af39318

fail 88% strict pass 45 pass · 6 fail

Run

Engine

ollama 7af39318 (7af393188defd52d370464de0d2064649cab9b41)

Run

2026-09-26T05:36:43Z to 2026-09-26T05:36:50Z

Platform

linux-x86_64, Python 3.12.3, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "go_version": "go1.26.0",
  "llama_cpp_build": "b11081",
  "llama_cpp_commit": "161755f",
  "pinned_version": "7af393188defd52d370464de0d2064649cab9b41",
  "runner": "llama-server token rendering (preserved_tokens) + model/parsers via server/routes.go flow",
  "tokens_per_step": "one"
}

Results file

data/ollama-7af39318.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Fail	Strict pass rate
`expected_match`	41	4	91%
`expected_error`	5	1	83%
`stream_equals_nonstream`	47	4	92%
`no_leakage`	51	0	100%
`arguments_json`	34	0	100%
`arguments_schema`	34	0	100%
`parallel_order`	0	1	0%

6 fixture(s) failed only a synthetic strategy: char:* (which can split special tokens), or a multi-token strategy this engine never produces because its server streams one token per event (one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8). These are listed below but never count toward the status.

Fixtures needing attention

fail gpt-oss/harmony-marker-in-arguments expected_match, stream_equals_nonstream

Checks that did not pass for gpt-oss/harmony-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'OllamaError: error parsing tool call: raw=\'{"path": "harmony.md", "content": "A call ends with <\|call\|>; a turn with <\|return\|>.\\nHeader: <\|start\|>assista...
`expected_match`	`token`	fail	parser raised 'OllamaError: error parsing tool call: raw=\'{"path": "harmony.md", "content": "A call ends with <\|call\|>; a turn with <\|return\|>.\\nHeader: <\|start\|>assista...
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream None, got 'Write the Harmony cheat sheet to a file.'

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-marker-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 7 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
-  "reasoning_content": "Write the Harmony cheat sheet to a file.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
-        "path": "harmony.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "exception": "OllamaError: error parsing tool call: raw='{\"path\": \"harmony.md\", \"content\": \"A call ends with <|call|>; a turn with <|return|>.\\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}', err=unexpected end of JSON input",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,6 @@
 {
   "content": null,
+  "exception": "OllamaError: error parsing tool call: raw='{\"path\": \"harmony.md\", \"content\": \"A call ends with <|call|>; a turn with <|return|>.\\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}', err=unexpected end of JSON input",
   "reasoning_content": "Write the Harmony cheat sheet to a file.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
-        "path": "harmony.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, reasoning, marker-in-arguments, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Write the Harmony cheat sheet to a file.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{"path": "harmony.md", "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>"}

Expected parse

{
  "content": null,
  "reasoning_content": "Write the Harmony cheat sheet to a file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
        "path": "harmony.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-marker-in-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Write the Harmony cheat sheet to a file.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"harmony.md\", \"content\": \"A call ends with <|call|>; a turn with <|return|>.\\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>\"}", "output_token_ids": [200005, 35644, 200008, 10930, 290, 102323, 65354, 16658, 316, 261, 1974, 13, 200007, 200006, 173781, 316, 28, 44580, 6765, 5933, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 4189, 1243, 392, 71, 90047, 44190, 672, 392, 3252, 1243, 392, 32, 2421, 17095, 483, 464, 91, 9925, 91, 170897, 261, 3716, 483, 464, 91, 1034, 91, 29, 15043, 77, 5737, 25, 464, 91, 5236, 91, 29, 173781, 27, 91, 21453, 91, 29, 12606, 815, 316, 28, 44580, 775, 6425, 464, 91, 542, 141043, 91, 29, 4108, 27, 91, 3938, 91, 11679, 157249, 91, 419, 91, 29, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Write the Harmony cheat sheet to a file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "harmony.md", "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>"}}]}, "tags": ["single-call", "reasoning", "marker-in-arguments", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). The special-token strings inside the argument value are ORDINARY text tokens in output_token_ids (openai-harmony encodes message content with special tokens disallowed). A parser that works on text rather than token ids cannot tell them from real markers."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

fail gpt-oss/harmony-truncated-in-arguments stream_equals_nonstream

Checks that did not pass for gpt-oss/harmony-truncated-in-arguments
Check	Strategy	Result	Detail
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream None, got 'The user asks for the weather in Paris. I should call get_weather.'

Synthetic strategies (not counted): stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-truncated-in-arguments --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 20 of fixtures/gpt-oss/rendered.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": null,
  "exception": "OllamaError: error parsing tool call: raw='{\"city\": \"Paris\", \"unit', err=unexpected end of JSON input",
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": []
}

Strategies: nonstream

{
  "content": null,
  "exception": "OllamaError: error parsing tool call: raw='{\"city\": \"Paris\", \"unit', err=unexpected end of JSON input",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: truncated, reasoning, x-recipient-in-role.

Expected graceful failure: Output stopped (max_tokens) inside the tool-call JSON; there is no complete call. (accept: no_tool_calls, content_passthrough, exception).

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris", "unit

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-truncated-in-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Output stopped (max_tokens) inside the tool-call JSON; there is no complete call.", "accept": ["no_tool_calls", "content_passthrough", "exception"]}, "tags": ["truncated", "reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

fail gpt-oss/llamacpp-builtin-python-recipient expected_match, stream_equals_nonstream

Checks that did not pass for gpt-oss/llamacpp-builtin-python-recipient
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	parser raised 'OllamaError: error parsing tool call: raw=\'print("hello")\', err=invalid character \'p\' looking for beginning of value'
`expected_match`	`token`	fail	parser raised 'OllamaError: error parsing tool call: raw=\'print("hello")\', err=invalid character \'p\' looking for beginning of value'
`stream_equals_nonstream`	`token`	fail	reasoning_content: nonstream None, got 'I will execute python to say hello'

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-builtin-python-recipient --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 16 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,5 +1,6 @@
 {
   "content": null,
-  "reasoning_content": "I will execute python to say hello",
+  "exception": "OllamaError: error parsing tool call: raw='print(\"hello\")', err=invalid character 'p' looking for beginning of value",
+  "reasoning_content": null,
   "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,5 @@
 {
   "content": null,
+  "exception": "OllamaError: error parsing tool call: raw='print(\"hello\")', err=invalid character 'p' looking for beginning of value",
   "reasoning_content": "I will execute python to say hello",
   "tool_calls": []

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6529-L6536. Tags: no-call, reasoning, x-builtin-recipient, x-recipient-in-channel.

Raw output

<|channel|>analysis<|message|>I will execute python to say hello<|end|><|start|>assistant<|channel|>commentary to=python <|constrain|>code<|message|>print("hello")

Expected parse

{
  "content": null,
  "reasoning_content": "I will execute python to say hello",
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-builtin-python-recipient", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6529-L6536", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>analysis<|message|>I will execute python to say hello<|end|><|start|>assistant<|channel|>commentary to=python <|constrain|>code<|message|>print(\"hello\")", "output_token_ids": [200005, 35644, 200008, 40, 738, 15792, 22752, 316, 2891, 40617, 200007, 200006, 173781, 200005, 12606, 815, 316, 28, 29010, 220, 200003, 3056, 200008, 1598, 568, 24912, 1405], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "I will execute python to say hello", "tool_calls": []}, "tags": ["no-call", "reasoning", "x-builtin-recipient", "x-recipient-in-channel"], "notes": "Unsolicited call to the built-in python tool, which was not offered: it is not a function tool call, and llama.cpp returns empty content."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

fail gpt-oss/llamacpp-stray-commentary-header expected_error, stream_equals_nonstream

Checks that did not pass for gpt-oss/llamacpp-stray-commentary-header
Check	Strategy	Result	Detail
`expected_error`	`nonstream`	fail	outcome exception "OllamaError: error parsing tool call: raw='I'm\nthinking', err=invalid character 'I' looking for beginning of value"; accepted: ['no_tool_calls']
`expected_error`	`token`	fail	outcome exception "OllamaError: error parsing tool call: raw='I'm\nthinking', err=invalid character 'I' looking for beginning of value"; accepted: ['no_tool_calls']
`stream_equals_nonstream`	`token`	fail	content: nonstream None, got "Hello, world!\nWhat's up?"

Synthetic strategies (not counted): expected_error@one fail, expected_error@special fail, expected_error@rand:1:8 fail, expected_error@rand:2:8 fail, expected_error@rand:3:8 fail, expected_error@rand:4:8 fail, expected_error@rand:5:8 fail, stream_equals_nonstream@one fail, stream_equals_nonstream@special fail, stream_equals_nonstream@rand:1:8 fail, stream_equals_nonstream@rand:2:8 fail, stream_equals_nonstream@rand:3:8 fail, stream_equals_nonstream@rand:4:8 fail, stream_equals_nonstream@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-stray-commentary-header --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 17 of fixtures/gpt-oss/imported.jsonl.

Observed

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

{
  "content": null,
  "exception": "OllamaError: error parsing tool call: raw='I'm\nthinking', err=invalid character 'I' looking for beginning of value",
  "reasoning_content": null,
  "tool_calls": []
}

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

{
  "content": "Hello, world!\nWhat's up?",
  "exception": "OllamaError: error parsing tool call: raw='I'm\nthinking', err=invalid character 'I' looking for beginning of value",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6540-L6546. Tags: no-call, reasoning, malformed, x-malformed-channel.

Expected graceful failure: Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\nthinking' and content 'Hello, world!\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error. (accept: no_tool_calls).

Raw output

<|channel|>commentary to=assistant<|channel|>analysis<|message|>I'm
thinking<|end|><|start|>assistant<|channel|>final<|message|>Hello, world!
What's up?

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-stray-commentary-header", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6540-L6546", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>commentary to=assistant<|channel|>analysis<|message|>I'm\nthinking<|end|><|start|>assistant<|channel|>final<|message|>Hello, world!\nWhat's up?", "output_token_ids": [200005, 12606, 815, 316, 28, 173781, 200005, 35644, 200008, 15390, 198, 149404, 200007, 200006, 173781, 200005, 17196, 200008, 13225, 11, 2375, 4175, 45350, 869, 30], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected_error": {"reason": "Stray '<|channel|>commentary to=assistant' header before the analysis channel. llama.cpp recovers reasoning 'I'm\\nthinking' and content 'Hello, world!\\nWhat's up?'; openai-harmony's non-strict parser keeps the content but drops the reasoning. Either way there is no tool call and the turn must not error.", "accept": ["no_tool_calls"]}, "tags": ["no-call", "reasoning", "malformed", "x-malformed-channel"], "notes": "Edge case '<|channel|>commentary to=assistant' before reasoning (the stray header llama.cpp #21286 tolerates)."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

fail gpt-oss/vllm-call-then-final expected_match

Checks that did not pass for gpt-oss/vllm-call-then-final
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Done', got None
`expected_match`	`token`	fail	content: expected 'Done', got None

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-call-then-final --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 7 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Done",
+  "content": null,
   "reasoning_content": "Reasoning about query...",
   "tool_calls": [

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718. Tags: single-call, reasoning, text-after-call, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{"query": "vllm"}<|call|><|start|>assistant<|channel|>final<|message|>Done

Expected parse

{
  "content": "Done",
  "reasoning_content": "Reasoning about query...",
  "tool_calls": [
    {
      "arguments": {
        "query": "vllm"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-call-then-final", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{\"query\": \"vllm\"}<|call|><|start|>assistant<|channel|>final<|message|>Done", "output_token_ids": [200005, 35644, 200008, 30377, 289, 1078, 5703, 1008, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 200003, 4108, 200008, 10848, 2975, 1243, 392, 85, 680, 76, 18583, 200012, 200006, 173781, 200005, 17196, 200008, 24537], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Done", "reasoning_content": "Reasoning about query...", "tool_calls": [{"name": "search", "arguments": {"query": "vllm"}}]}, "tags": ["single-call", "reasoning", "text-after-call", "x-recipient-in-role"], "notes": "test_cross_channel_with_tool. As in vllm-sequential-calls, text after <|call|> is only seen when the stop token is not applied."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

fail gpt-oss/vllm-sequential-calls expected_match, parallel_order

Checks that did not pass for gpt-oss/vllm-sequential-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	tool_calls: expected ['get_weather', 'get_time'], got ['get_weather']
`expected_match`	`token`	fail	tool_calls: expected ['get_weather', 'get_time'], got ['get_weather']
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'get_time'], got 1 ['get_weather']
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'get_time'], got 1 ['get_weather']

Synthetic strategies (not counted): expected_match@one fail, expected_match@special fail, expected_match@rand:1:8 fail, expected_match@rand:2:8 fail, expected_match@rand:3:8 fail, expected_match@rand:4:8 fail, expected_match@rand:5:8 fail, parallel_order@one fail, parallel_order@special fail, parallel_order@rand:1:8 fail, parallel_order@rand:2:8 fail, parallel_order@rand:3:8 fail, parallel_order@rand:4:8 fail, parallel_order@rand:5:8 fail

Minimal repro

uv run canitoolcall run --engine ollama --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-sequential-calls --strategy token --observed all

Set up the engine first with scripts/engines/ollama.sh; this run used ollama 7af39318. The fixture is line 6 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -8,10 +8,4 @@
       },
       "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "timezone": "UTC"
-      },
-      "name": "get_time"
     }
   ]

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745. Tags: parallel-calls, reasoning, x-sequential-calls, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{"location": "Paris"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{"timezone": "UTC"}

Expected parse

{
  "content": null,
  "reasoning_content": "Thinking",
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "timezone": "UTC"
      },
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-sequential-calls", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {"type": "object", "properties": {"timezone": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{\"timezone\": \"UTC\"}", "output_token_ids": [200005, 35644, 200008, 133850, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200003, 4108, 200008, 10848, 7693, 1243, 392, 72782, 18583, 200012, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 200003, 4108, 200008, 10848, 106775, 1243, 392, 32674, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Thinking", "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}, {"name": "get_time", "arguments": {"timezone": "UTC"}}]}, "tags": ["parallel-calls", "reasoning", "x-sequential-calls", "x-recipient-in-role"], "notes": "test_tool_index_across_calls: the two deltas concatenated. Harmony has no parallel-call wrapper; consecutive call messages are separated by <|call|>, which is a stop token, so this output only reaches a parser when the stop is not applied (e.g. ignore_eos, vLLM #50690). The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "chunking": "one llama-server event per token; Add(delta, false) per non-empty group, then Add('', true)",
  "detokenizer": "llama-server common_token_to_piece(special = id in preserved_tokens)",
  "eog_stop": "tokens after the first end-of-generation token are dropped (llama-server stops there)",
  "has_thinking_support": true,
  "legacy_template_parser": false,
  "llama_cpp_build": "b11081",
  "nonstream": "per-token events concatenated (routes.go writeChatResponse)",
  "ollama_commit": "7af393188defd52d370464de0d2064649cab9b41",
  "parser": "harmony",
  "parser_source": "server/routes.go shouldUseHarmony: gptoss family -> harmony",
  "preserved_tokens": [
    "<|start|>",
    "<|end|>",
    "<|message|>",
    "<|channel|>",
    "<|constrain|>"
  ],
  "think_effective": true,
  "think_requested": null,
  "tool_call_serialization": "openai.ToToolCalls",
  "vocab_gguf": "openai--gpt-oss-20b.vocab.gguf",
  "vocab_gguf_converter_llama_cpp_commit": "a25c9865fe03c954c93fd755b5d79ae86ba99750",
  "vocab_gguf_revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
  "vocab_gguf_revision_matches_tokenizer": true,
  "vocab_gguf_sha256": "41ab8478c7fafbb0012cdeda945237760bc83660a4c1c2772de03e0c867e5be9"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model