gpt-oss (Harmony response format) on sglang 0.5.20

fail 37% strict pass 19 pass · 32 fail

Run

Engine

sglang 0.5.20

Run

2026-09-26T05:19:05Z to 2026-09-26T05:22:38Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1)

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "deps": {
    "orjson": "3.12.0",
    "partial-json-parser": "0.2.1.1.post7",
    "pydantic": "2.13.5",
    "tokenizers": "0.22.2",
    "torch": "2.13.0",
    "transformers": "5.12.1",
    "xgrammar": "0.2.1"
  },
  "dist_info": "sglang-0.5.20.dist-info",
  "install": "cp312 manylinux wheel unpacked and added via .pth (scripts/engines/sglang.sh)",
  "pinned_version": "0.5.20",
  "reasoning_parsers": 31,
  "tokens_per_step": "many",
  "tool_call_parsers": 39
}

Results file

data/sglang-0.5.20.json

Checks

Per fixture, the worst result of each check over the non-streaming parse and every realistic chunking strategy.
Check	Pass	Fail	Strict pass rate
`expected_match`	13	32	29%
`expected_error`	6	0	100%
`stream_equals_nonstream`	21	30	41%
`split_invariance`	41	10	80%
`no_leakage`	22	29	43%
`arguments_json`	6	0	100%
`arguments_schema`	6	0	100%
`parallel_order`	0	1	0%

Fixtures needing attention

fail gpt-oss/harmony-compact-json expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-compact-json
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city":"San Francisco"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-compact-json --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\":\"San Francisco\"}<|call|>",
   "reasoning_content": "Need to use function get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "San Francisco"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "Need to use function get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "San Francisco"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, reasoning, x-recipient-in-role, x-constrain-token.

Raw output

<|channel|>analysis<|message|>Need to use function get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city":"San Francisco"}

Expected parse

{
  "content": null,
  "reasoning_content": "Need to use function get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "San Francisco"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-compact-json", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Need to use function get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\":\"San Francisco\"}", "output_token_ids": [200005, 35644, 200008, 23483, 316, 1199, 1114, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 7534, 28499, 18826, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Need to use function get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "San Francisco"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-constrain-token"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). Arguments without whitespace, as in the Harmony spec's own examples."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-content-type-json-text expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-content-type-json-text
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-content-type-json-text --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 18 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{\"city\": \"Paris\"}<|call|>",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{"city": "Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-content-type-json-text", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{\"city\": \"Paris\"}", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 5701, 200008, 10848, 17500, 1243, 392, 72782, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-constrain-text"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). Content type written as plain ' json' (no <|constrain|> token), rendered by openai-harmony with content_type='json'. The HF chat template uses this spelling; vLLM's harmony parser accepts both."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-empty-arguments expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>'; tool_calls: expected ['get_time'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_time'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary <\|constrain\|>json<\|message\|>{}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_time<|channel|>commentary <|constrain|>json<|message|>{}<|call|>",
   "reasoning_content": "Get the time.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": "Get the time.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Get the time.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary <|constrain|>json<|message|>{}

Expected parse

{
  "content": null,
  "reasoning_content": "Get the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-empty-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Get the time.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary <|constrain|>json<|message|>{}", "output_token_ids": [200005, 35644, 200008, 1816, 290, 1058, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 220, 200003, 4108, 200008, 12083], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Get the time.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "reasoning", "empty-arguments", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-long-arguments expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-long-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ...; tool_calls: expected ['write_file'], got []
`expected_match`	`one`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`special`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`token`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['write_file'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "steps.py", "content": "\\"\\"\\"Generated module used ..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-long-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 16 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"steps.py\", \"content\": \"\\\"\\\"\\\"Generated module used as a long tool argument.\\\"\\\"\\\"\\n\\nfrom __future__ import annotations\\n\\ndef step_00(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 0.\\\"\\\"\\\"\\n    return x + 0\\n\\ndef step_01(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 1.\\\"\\\"\\\"\\n    return x + 1\\n\\ndef step_02(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 2.\\\"\\\"\\\"\\n    return x + 2\\n\\ndef step_03(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 3.\\\"\\\"\\\"\\n    return x + 3\\n\\ndef step_04(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 4.\\\"\\\"\\\"\\n    return x + 4\\n\\ndef step_05(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 5.\\\"\\\"\\\"\\n    return x + 5\\n\\ndef step_06(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 6.\\\"\\\"\\\"\\n    return x + 6\\n\\ndef step_07(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 7.\\\"\\\"\\\"\\n    return x + 7\\n\\ndef step_08(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 8.\\\"\\\"\\\"\\n    return x + 8\\n\\ndef step_09(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 9.\\\"\\\"\\\"\\n    return x + 9\\n\\ndef step_10(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 10.\\\"\\\"\\\"\\n    return x + 10\\n\\ndef step_11(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 11.\\\"\\\"\\\"\\n    return x + 11\\n\\ndef step_12(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 12.\\\"\\\"\\\"\\n    return x + 12\\n\\ndef step_13(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 13.\\\"\\\"\\\"\\n    return x + 13\\n\\ndef step_14(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 14.\\\"\\\"\\\"\\n    return x + 14\\n\\ndef step_15(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 15.\\\"\\\"\\\"\\n    return x + 15\\n\\ndef step_16(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 16.\\\"\\\"\\\"\\n    return x + 16\\n\\ndef step_17(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 17.\\\"\\\"\\\"\\n    return x + 17\\n\\ndef step_18(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 18.\\\"\\\"\\\"\\n    return x + 18\\n\\ndef step_19(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 19.\\\"\\\"\\\"\\n    return x + 19\\n\\ndef step_20(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 20.\\\"\\\"\\\"\\n    return x + 20\\n\\ndef step_21(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 21.\\\"\\\"\\\"\\n    return x + 21\\n\\ndef step_22(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 22.\\\"\\\"\\\"\\n    return x + 22\\n\\ndef step_23(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 23.\\\"\\\"\\\"\\n    return x + 23\\n\\ndef step_24(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 24.\\\"\\\"\\\"\\n    return x + 24\\n\\ndef step_25(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 25.\\\"\\\"\\\"\\n    return x + 25\\n\\ndef step_26(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 26.\\\"\\\"\\\"\\n    return x + 26\\n\\ndef step_27(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 27.\\\"\\\"\\\"\\n    return x + 27\\n\\ndef step_28(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 28.\\\"\\\"\\\"\\n    return x + 28\\n\\ndef step_29(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 29.\\\"\\\"\\\"\\n    return x + 29\\n\\ndef step_30(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 30.\\\"\\\"\\\"\\n    return x + 30\\n\\ndef step_31(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 31.\\\"\\\"\\\"\\n    return x + 31\\n\\ndef step_32(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 32.\\\"\\\"\\\"\\n    return x + 32\\n\\ndef step_33(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 33.\\\"\\\"\\\"\\n    return x + 33\\n\\ndef step_34(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 34.\\\"\\\"\\\"\\n    return x + 34\\n\\ndef step_35(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 35.\\\"\\\"\\\"\\n    return x + 35\\n\\ndef step_36(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 36.\\\"\\\"\\\"\\n    return x + 36\\n\\ndef step_37(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 37.\\\"\\\"\\\"\\n    return x + 37\\n\\ndef step_38(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 38.\\\"\\\"\\\"\\n    return x + 38\\n\\ndef step_39(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 39.\\\"\\\"\\\"\\n    return x + 39\\n\\ndef step_40(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 40.\\\"\\\"\\\"\\n    return x + 40\\n\\ndef step_41(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 41.\\\"\\\"\\\"\\n    return x + 41\\n\\ndef step_42(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 42.\\\"\\\"\\\"\\n    return x + 42\\n\\ndef step_43(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 43.\\\"\\\"\\\"\\n    return x + 43\\n\\ndef step_44(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 44.\\\"\\\"\\\"\\n    return x + 44\\n\\ndef step_45(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 45.\\\"\\\"\\\"\\n    return x + 45\\n\\ndef step_46(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 46.\\\"\\\"\\\"\\n    return x + 46\\n\\ndef step_47(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 47.\\\"\\\"\\\"\\n    return x + 47\\n\\ndef step_48(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 48.\\\"\\\"\\\"\\n    return x + 48\\n\\ndef step_49(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 49.\\\"\\\"\\\"\\n    return x + 49\\n\\ndef step_50(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 50.\\\"\\\"\\\"\\n    return x + 50\\n\\ndef step_51(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 51.\\\"\\\"\\\"\\n    return x + 51\\n\\ndef step_52(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 52.\\\"\\\"\\\"\\n    return x + 52\\n\\ndef step_53(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 53.\\\"\\\"\\\"\\n    return x + 53\\n\\ndef step_54(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 54.\\\"\\\"\\\"\\n    return x + 54\\n\\ndef step_55(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 55.\\\"\\\"\\\"\\n    return x + 55\\n\\ndef step_56(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 56.\\\"\\\"\\\"\\n    return x + 56\\n\\ndef step_57(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 57.\\\"\\\"\\\"\\n    return x + 57\\n\\ndef step_58(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 58.\\\"\\\"\\\"\\n    return x + 58\\n\\ndef step_59(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 59.\\\"\\\"\\\"\\n    return x + 59\\n\"}<|call|>",
   "reasoning_content": "Write the module.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    \"\"\"Return x plus 0.\"\"\"\n    return x + 0\n\ndef step_01(x: int) -> int:\n    \"\"\"Return x plus 1.\"\"\"\n    return x + 1\n\ndef step_02(x: int) -> int:\n    \"\"\"Return x plus 2.\"\"\"\n    return x + 2\n\ndef step_03(x: int) -> int:\n    \"\"\"Return x plus 3.\"\"\"\n    return x + 3\n\ndef step_04(x: int) -> int:\n    \"\"\"Return x plus 4.\"\"\"\n    return x + 4\n\ndef step_05(x: int) -> int:\n    \"\"\"Return x plus 5.\"\"\"\n    return x + 5\n\ndef step_06(x: int) -> int:\n    \"\"\"Return x plus 6.\"\"\"\n    return x + 6\n\ndef step_07(x: int) -> int:\n    \"\"\"Return x plus 7.\"\"\"\n    return x + 7\n\ndef step_08(x: int) -> int:\n    \"\"\"Return x plus 8.\"\"\"\n    return x + 8\n\ndef step_09(x: int) -> int:\n    \"\"\"Return x plus 9.\"\"\"\n    return x + 9\n\ndef step_10(x: int) -> int:\n    \"\"\"Return x plus 10.\"\"\"\n    return x + 10\n\ndef step_11(x: int) -> int:\n    \"\"\"Return x plus 11.\"\"\"\n    return x + 11\n\ndef step_12(x: int) -> int:\n    \"\"\"Return x plus 12.\"\"\"\n    return x + 12\n\ndef step_13(x: int) -> int:\n    \"\"\"Return x plus 13.\"\"\"\n    return x + 13\n\ndef step_14(x: int) -> int:\n    \"\"\"Return x plus 14.\"\"\"\n    return x + 14\n\ndef step_15(x: int) -> int:\n    \"\"\"Return x plus 15.\"\"\"\n    return x + 15\n\ndef step_16(x: int) -> int:\n    \"\"\"Return x plus 16.\"\"\"\n    return x + 16\n\ndef step_17(x: int) -> int:\n    \"\"\"Return x plus 17.\"\"\"\n    return x + 17\n\ndef step_18(x: int) -> int:\n    \"\"\"Return x plus 18.\"\"\"\n    return x + 18\n\ndef step_19(x: int) -> int:\n    \"\"\"Return x plus 19.\"\"\"\n    return x + 19\n\ndef step_20(x: int) -> int:\n    \"\"\"Return x plus 20.\"\"\"\n    return x + 20\n\ndef step_21(x: int) -> int:\n    \"\"\"Return x plus 21.\"\"\"\n    return x + 21\n\ndef step_22(x: int) -> int:\n    \"\"\"Return x plus 22.\"\"\"\n    return x + 22\n\ndef step_23(x: int) -> int:\n    \"\"\"Return x plus 23.\"\"\"\n    return x + 23\n\ndef step_24(x: int) -> int:\n    \"\"\"Return x plus 24.\"\"\"\n    return x + 24\n\ndef step_25(x: int) -> int:\n    \"\"\"Return x plus 25.\"\"\"\n    return x + 25\n\ndef step_26(x: int) -> int:\n    \"\"\"Return x plus 26.\"\"\"\n    return x + 26\n\ndef step_27(x: int) -> int:\n    \"\"\"Return x plus 27.\"\"\"\n    return x + 27\n\ndef step_28(x: int) -> int:\n    \"\"\"Return x plus 28.\"\"\"\n    return x + 28\n\ndef step_29(x: int) -> int:\n    \"\"\"Return x plus 29.\"\"\"\n    return x + 29\n\ndef step_30(x: int) -> int:\n    \"\"\"Return x plus 30.\"\"\"\n    return x + 30\n\ndef step_31(x: int) -> int:\n    \"\"\"Return x plus 31.\"\"\"\n    return x + 31\n\ndef step_32(x: int) -> int:\n    \"\"\"Return x plus 32.\"\"\"\n    return x + 32\n\ndef step_33(x: int) -> int:\n    \"\"\"Return x plus 33.\"\"\"\n    return x + 33\n\ndef step_34(x: int) -> int:\n    \"\"\"Return x plus 34.\"\"\"\n    return x + 34\n\ndef step_35(x: int) -> int:\n    \"\"\"Return x plus 35.\"\"\"\n    return x + 35\n\ndef step_36(x: int) -> int:\n    \"\"\"Return x plus 36.\"\"\"\n    return x + 36\n\ndef step_37(x: int) -> int:\n    \"\"\"Return x plus 37.\"\"\"\n    return x + 37\n\ndef step_38(x: int) -> int:\n    \"\"\"Return x plus 38.\"\"\"\n    return x + 38\n\ndef step_39(x: int) -> int:\n    \"\"\"Return x plus 39.\"\"\"\n    return x + 39\n\ndef step_40(x: int) -> int:\n    \"\"\"Return x plus 40.\"\"\"\n    return x + 40\n\ndef step_41(x: int) -> int:\n    \"\"\"Return x plus 41.\"\"\"\n    return x + 41\n\ndef step_42(x: int) -> int:\n    \"\"\"Return x plus 42.\"\"\"\n    return x + 42\n\ndef step_43(x: int) -> int:\n    \"\"\"Return x plus 43.\"\"\"\n    return x + 43\n\ndef step_44(x: int) -> int:\n    \"\"\"Return x plus 44.\"\"\"\n    return x + 44\n\ndef step_45(x: int) -> int:\n    \"\"\"Return x plus 45.\"\"\"\n    return x + 45\n\ndef step_46(x: int) -> int:\n    \"\"\"Return x plus 46.\"\"\"\n    return x + 46\n\ndef step_47(x: int) -> int:\n    \"\"\"Return x plus 47.\"\"\"\n    return x + 47\n\ndef step_48(x: int) -> int:\n    \"\"\"Return x plus 48.\"\"\"\n    return x + 48\n\ndef step_49(x: int) -> int:\n    \"\"\"Return x plus 49.\"\"\"\n    return x + 49\n\ndef step_50(x: int) -> int:\n    \"\"\"Return x plus 50.\"\"\"\n    return x + 50\n\ndef step_51(x: int) -> int:\n    \"\"\"Return x plus 51.\"\"\"\n    return x + 51\n\ndef step_52(x: int) -> int:\n    \"\"\"Return x plus 52.\"\"\"\n    return x + 52\n\ndef step_53(x: int) -> int:\n    \"\"\"Return x plus 53.\"\"\"\n    return x + 53\n\ndef step_54(x: int) -> int:\n    \"\"\"Return x plus 54.\"\"\"\n    return x + 54\n\ndef step_55(x: int) -> int:\n    \"\"\"Return x plus 55.\"\"\"\n    return x + 55\n\ndef step_56(x: int) -> int:\n    \"\"\"Return x plus 56.\"\"\"\n    return x + 56\n\ndef step_57(x: int) -> int:\n    \"\"\"Return x plus 57.\"\"\"\n    return x + 57\n\ndef step_58(x: int) -> int:\n    \"\"\"Return x plus 58.\"\"\"\n    return x + 58\n\ndef step_59(x: int) -> int:\n    \"\"\"Return x plus 59.\"\"\"\n    return x + 59\n",
-        "path": "steps.py"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": "Write the module.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    \"\"\"Return x plus 0.\"\"\"\n    return x + 0\n\ndef step_01(x: int) -> int:\n    \"\"\"Return x plus 1.\"\"\"\n    return x + 1\n\ndef step_02(x: int) -> int:\n    \"\"\"Return x plus 2.\"\"\"\n    return x + 2\n\ndef step_03(x: int) -> int:\n    \"\"\"Return x plus 3.\"\"\"\n    return x + 3\n\ndef step_04(x: int) -> int:\n    \"\"\"Return x plus 4.\"\"\"\n    return x + 4\n\ndef step_05(x: int) -> int:\n    \"\"\"Return x plus 5.\"\"\"\n    return x + 5\n\ndef step_06(x: int) -> int:\n    \"\"\"Return x plus 6.\"\"\"\n    return x + 6\n\ndef step_07(x: int) -> int:\n    \"\"\"Return x plus 7.\"\"\"\n    return x + 7\n\ndef step_08(x: int) -> int:\n    \"\"\"Return x plus 8.\"\"\"\n    return x + 8\n\ndef step_09(x: int) -> int:\n    \"\"\"Return x plus 9.\"\"\"\n    return x + 9\n\ndef step_10(x: int) -> int:\n    \"\"\"Return x plus 10.\"\"\"\n    return x + 10\n\ndef step_11(x: int) -> int:\n    \"\"\"Return x plus 11.\"\"\"\n    return x + 11\n\ndef step_12(x: int) -> int:\n    \"\"\"Return x plus 12.\"\"\"\n    return x + 12\n\ndef step_13(x: int) -> int:\n    \"\"\"Return x plus 13.\"\"\"\n    return x + 13\n\ndef step_14(x: int) -> int:\n    \"\"\"Return x plus 14.\"\"\"\n    return x + 14\n\ndef step_15(x: int) -> int:\n    \"\"\"Return x plus 15.\"\"\"\n    return x + 15\n\ndef step_16(x: int) -> int:\n    \"\"\"Return x plus 16.\"\"\"\n    return x + 16\n\ndef step_17(x: int) -> int:\n    \"\"\"Return x plus 17.\"\"\"\n    return x + 17\n\ndef step_18(x: int) -> int:\n    \"\"\"Return x plus 18.\"\"\"\n    return x + 18\n\ndef step_19(x: int) -> int:\n    \"\"\"Return x plus 19.\"\"\"\n    return x + 19\n\ndef step_20(x: int) -> int:\n    \"\"\"Return x plus 20.\"\"\"\n    return x + 20\n\ndef step_21(x: int) -> int:\n    \"\"\"Return x plus 21.\"\"\"\n    return x + 21\n\ndef step_22(x: int) -> int:\n    \"\"\"Return x plus 22.\"\"\"\n    return x + 22\n\ndef step_23(x: int) -> int:\n    \"\"\"Return x plus 23.\"\"\"\n    return x + 23\n\ndef step_24(x: int) -> int:\n    \"\"\"Return x plus 24.\"\"\"\n    return x + 24\n\ndef step_25(x: int) -> int:\n    \"\"\"Return x plus 25.\"\"\"\n    return x + 25\n\ndef step_26(x: int) -> int:\n    \"\"\"Return x plus 26.\"\"\"\n    return x + 26\n\ndef step_27(x: int) -> int:\n    \"\"\"Return x plus 27.\"\"\"\n    return x + 27\n\ndef step_28(x: int) -> int:\n    \"\"\"Return x plus 28.\"\"\"\n    return x + 28\n\ndef step_29(x: int) -> int:\n    \"\"\"Return x plus 29.\"\"\"\n    return x + 29\n\ndef step_30(x: int) -> int:\n    \"\"\"Return x plus 30.\"\"\"\n    return x + 30\n\ndef step_31(x: int) -> int:\n    \"\"\"Return x plus 31.\"\"\"\n    return x + 31\n\ndef step_32(x: int) -> int:\n    \"\"\"Return x plus 32.\"\"\"\n    return x + 32\n\ndef step_33(x: int) -> int:\n    \"\"\"Return x plus 33.\"\"\"\n    return x + 33\n\ndef step_34(x: int) -> int:\n    \"\"\"Return x plus 34.\"\"\"\n    return x + 34\n\ndef step_35(x: int) -> int:\n    \"\"\"Return x plus 35.\"\"\"\n    return x + 35\n\ndef step_36(x: int) -> int:\n    \"\"\"Return x plus 36.\"\"\"\n    return x + 36\n\ndef step_37(x: int) -> int:\n    \"\"\"Return x plus 37.\"\"\"\n    return x + 37\n\ndef step_38(x: int) -> int:\n    \"\"\"Return x plus 38.\"\"\"\n    return x + 38\n\ndef step_39(x: int) -> int:\n    \"\"\"Return x plus 39.\"\"\"\n    return x + 39\n\ndef step_40(x: int) -> int:\n    \"\"\"Return x plus 40.\"\"\"\n    return x + 40\n\ndef step_41(x: int) -> int:\n    \"\"\"Return x plus 41.\"\"\"\n    return x + 41\n\ndef step_42(x: int) -> int:\n    \"\"\"Return x plus 42.\"\"\"\n    return x + 42\n\ndef step_43(x: int) -> int:\n    \"\"\"Return x plus 43.\"\"\"\n    return x + 43\n\ndef step_44(x: int) -> int:\n    \"\"\"Return x plus 44.\"\"\"\n    return x + 44\n\ndef step_45(x: int) -> int:\n    \"\"\"Return x plus 45.\"\"\"\n    return x + 45\n\ndef step_46(x: int) -> int:\n    \"\"\"Return x plus 46.\"\"\"\n    return x + 46\n\ndef step_47(x: int) -> int:\n    \"\"\"Return x plus 47.\"\"\"\n    return x + 47\n\ndef step_48(x: int) -> int:\n    \"\"\"Return x plus 48.\"\"\"\n    return x + 48\n\ndef step_49(x: int) -> int:\n    \"\"\"Return x plus 49.\"\"\"\n    return x + 49\n\ndef step_50(x: int) -> int:\n    \"\"\"Return x plus 50.\"\"\"\n    return x + 50\n\ndef step_51(x: int) -> int:\n    \"\"\"Return x plus 51.\"\"\"\n    return x + 51\n\ndef step_52(x: int) -> int:\n    \"\"\"Return x plus 52.\"\"\"\n    return x + 52\n\ndef step_53(x: int) -> int:\n    \"\"\"Return x plus 53.\"\"\"\n    return x + 53\n\ndef step_54(x: int) -> int:\n    \"\"\"Return x plus 54.\"\"\"\n    return x + 54\n\ndef step_55(x: int) -> int:\n    \"\"\"Return x plus 55.\"\"\"\n    return x + 55\n\ndef step_56(x: int) -> int:\n    \"\"\"Return x plus 56.\"\"\"\n    return x + 56\n\ndef step_57(x: int) -> int:\n    \"\"\"Return x plus 57.\"\"\"\n    return x + 57\n\ndef step_58(x: int) -> int:\n    \"\"\"Return x plus 58.\"\"\"\n    return x + 58\n\ndef step_59(x: int) -> int:\n    \"\"\"Return x plus 59.\"\"\"\n    return x + 59\n",
-        "path": "steps.py"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Write the module.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{"path": "steps.py", "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    \"\"\"Return x plus 0.\"\"\"\n    return x + 0\n\ndef step_01(x: int) -> int:\n    \"\"\"Return x plus 1.\"\"\"\n    return x + 1\n\ndef step_02(x: int) -> int:\n    \"\"\"Return x plus 2.\"\"\"\n    return x + 2\n\ndef step_03(x: int) -> int:\n    \"\"\"Return x plus 3.\"\"\"\n    return x + 3\n\ndef step_04(x: int) -> int:\n    \"\"\"Return x plus 4.\"\"\"\n    return x + 4\n\ndef step_05(x: int) -> int:\n    \"\"\"Return x plus 5.\"\"\"\n    return x + 5\n\ndef step_06(x: int) -> int:\n    \"\"\"Return x plus 6.\"\"\"\n    return x + 6\n\ndef step_07(x: int) -> int:\n    \"\"\"Return x plus 7.\"\"\"\n    return x + 7\n\ndef step_08(x: int) -> int:\n    \"\"\"Return x plus 8.\"\"\"\n    return x + 8\n\ndef step_09(x: int) -> int:\n    \"\"\"Return x plus 9.\"\"\"\n    return x + 9\n\ndef step_10(x: int) -> int:\n    \"\"\"Return x plus 10.\"\"\"\n    return x + 10\n\ndef step_11(x: int) -> int:\n    \"\"\"Return x plus 11.\"\"\"\n    return x + 11\n\ndef step_12(x: int) -> int:\n    \"\"\"Return x plus 12.\"\"\"\n    return x + 12\n\ndef step_13(x: int) -> int:\n    \"\"\"Return x plus 13.\"\"\"\n    return x + 13\n\ndef step_14(x: int) -> int:\n    \"\"\"Return x plus 14.\"\"\"\n    return x + 14\n\ndef step_15(x: int) -> int:\n    \"\"\"Return x plus 15.\"\"\"\n    return x + 15\n\ndef step_16(x: int) -> int:\n    \"\"\"Return x plus 16.\"\"\"\n    return x + 16\n\ndef step_17(x: int) -> int:\n    \"\"\"Return x plus 17.\"\"\"\n    return x + 17\n\ndef step_18(x: int) -> int:\n    \"\"\"Return x plus 18.\"\"\"\n    return x + 18\n\ndef step_19(x: int) -> int:\n    \"\"\"Return x plus 19.\"\"\"\n    return x + 19\n\ndef step_20(x: int) -> int:\n    \"\"\"Return x plus 20.\"\"\"\n    return x + 20\n\ndef step_21(x: int) -> int:\n    \"\"\"Return x plus 21.\"\"\"\n    return x + 21\n\ndef step_22(x: int) -> int:\n    \"\"\"Return x plus 22.\"\"\"\n    return x + 22\n\ndef step_23(x: int) -> int:\n    \"\"\"Return x plus 23.\"\"\"\n    return x + 23\n\ndef step_24(x: int) -> int:\n    \"\"\"Return x plus 24.\"\"\"\n    return x + 24\n\ndef step_25(x: int) -> int:\n    \"\"\"Return x plus 25.\"\"\"\n    return x + 25\n\ndef step_26(x: int) -> int:\n    \"\"\"Return x plus 26.\"\"\"\n    return x + 26\n\ndef step_27(x: int) -> int:\n    \"\"\"Return x plus 27.\"\"\"\n    return x + 27\n\ndef step_28(x: int) -> int:\n    \"\"\"Return x plus 28.\"\"\"\n    return x + 28\n\ndef step_29(x: int) -> int:\n    \"\"\"Return x plus 29.\"\"\"\n    return x + 29\n\ndef step_30(x: int) -> int:\n    \"\"\"Return x plus 30.\"\"\"\n    return x + 30\n\ndef step_31(x: int) -> int:\n    \"\"\"Return x plus 31.\"\"\"\n    return x + 31\n\ndef step_32(x: int) -> int:\n    \"\"\"Return x plus 32.\"\"\"\n    return x + 32\n\ndef step_33(x: int) -> int:\n    \"\"\"Return x plus 33.\"\"\"\n    return x + 33\n\ndef step_34(x: int) -> int:\n    \"\"\"Return x plus 34.\"\"\"\n    return x + 34\n\ndef step_35(x: int) -> int:\n    \"\"\"Return x plus 35.\"\"\"\n    return x + 35\n\ndef step_36(x: int) -> int:\n    \"\"\"Return x plus 36.\"\"\"\n    return x + 36\n\ndef step_37(x: int) -> int:\n    \"\"\"Return x plus 37.\"\"\"\n    return x + 37\n\ndef step_38(x: int) -> int:\n    \"\"\"Return x plus 38.\"\"\"\n    return x + 38\n\ndef step_39(x: int) -> int:\n    \"\"\"Return x plus 39.\"\"\"\n    return x + 39\n\ndef step_40(x: int) -> int:\n    \"\"\"Return x plus 40.\"\"\"\n    return x + 40\n\ndef step_41(x: int) -> int:\n    \"\"\"Return x plus 41.\"\"\"\n    return x + 41\n\ndef step_42(x: int) -> int:\n    \"\"\"Return x plus 42.\"\"\"\n    return x + 42\n\ndef step_43(x: int) -> int:\n    \"\"\"Return x plus 43.\"\"\"\n    return x + 43\n\ndef step_44(x: int) -> int:\n    \"\"\"Return x plus 44.\"\"\"\n    return x + 44\n\ndef step_45(x: int) -> int:\n    \"\"\"Return x plus 45.\"\"\"\n    return x + 45\n\ndef step_46(x: int) -> int:\n    \"\"\"Return x plus 46.\"\"\"\n    return x + 46\n\ndef step_47(x: int) -> int:\n    \"\"\"Return x plus 47.\"\"\"\n    return x + 47\n\ndef step_48(x: int) -> int:\n    \"\"\"Return x plus 48.\"\"\"\n    return x + 48\n\ndef step_49(x: int) -> int:\n    \"\"\"Return x plus 49.\"\"\"\n    return x + 49\n\ndef step_50(x: int) -> int:\n    \"\"\"Return x plus 50.\"\"\"\n    return x + 50\n\ndef step_51(x: int) -> int:\n    \"\"\"Return x plus 51.\"\"\"\n    return x + 51\n\ndef step_52(x: int) -> int:\n    \"\"\"Return x plus 52.\"\"\"\n    return x + 52\n\ndef step_53(x: int) -> int:\n    \"\"\"Return x plus 53.\"\"\"\n    return x + 53\n\ndef step_54(x: int) -> int:\n    \"\"\"Return x plus 54.\"\"\"\n    return x + 54\n\ndef step_55(x: int) -> int:\n    \"\"\"Return x plus 55.\"\"\"\n    return x + 55\n\ndef step_56(x: int) -> int:\n    \"\"\"Return x plus 56.\"\"\"\n    return x + 56\n\ndef step_57(x: int) -> int:\n    \"\"\"Return x plus 57.\"\"\"\n    return x + 57\n\ndef step_58(x: int) -> int:\n    \"\"\"Return x plus 58.\"\"\"\n    return x + 58\n\ndef step_59(x: int) -> int:\n    \"\"\"Return x plus 59.\"\"\"\n    return x + 59\n"}

Expected parse

{
  "content": null,
  "reasoning_content": "Write the module.",
  "tool_calls": [
    {
      "arguments": {
        "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    \"\"\"Return x plus 0.\"\"\"\n    return x + 0\n\ndef step_01(x: int) -> int:\n    \"\"\"Return x plus 1.\"\"\"\n    return x + 1\n\ndef step_02(x: int) -> int:\n    \"\"\"Return x plus 2.\"\"\"\n    return x + 2\n\ndef step_03(x: int) -> int:\n    \"\"\"Return x plus 3.\"\"\"\n    return x + 3\n\ndef step_04(x: int) -> int:\n    \"\"\"Return x plus 4.\"\"\"\n    return x + 4\n\ndef step_05(x: int) -> int:\n    \"\"\"Return x plus 5.\"\"\"\n    return x + 5\n\ndef step_06(x: int) -> int:\n    \"\"\"Return x plus 6.\"\"\"\n    return x + 6\n\ndef step_07(x: int) -> int:\n    \"\"\"Return x plus 7.\"\"\"\n    return x + 7\n\ndef step_08(x: int) -> int:\n    \"\"\"Return x plus 8.\"\"\"\n    return x + 8\n\ndef step_09(x: int) -> int:\n    \"\"\"Return x plus 9.\"\"\"\n    return x + 9\n\ndef step_10(x: int) -> int:\n    \"\"\"Return x plus 10.\"\"\"\n    return x + 10\n\ndef step_11(x: int) -> int:\n    \"\"\"Return x plus 11.\"\"\"\n    return x + 11\n\ndef step_12(x: int) -> int:\n    \"\"\"Return x plus 12.\"\"\"\n    return x + 12\n\ndef step_13(x: int) -> int:\n    \"\"\"Return x plus 13.\"\"\"\n    return x + 13\n\ndef step_14(x: int) -> int:\n    \"\"\"Return x plus 14.\"\"\"\n    return x + 14\n\ndef step_15(x: int) -> int:\n    \"\"\"Return x plus 15.\"\"\"\n    return x + 15\n\ndef step_16(x: int) -> int:\n    \"\"\"Return x plus 16.\"\"\"\n    return x + 16\n\ndef step_17(x: int) -> int:\n    \"\"\"Return x plus 17.\"\"\"\n    return x + 17\n\ndef step_18(x: int) -> int:\n    \"\"\"Return x plus 18.\"\"\"\n    return x + 18\n\ndef step_19(x: int) -> int:\n    \"\"\"Return x plus 19.\"\"\"\n    return x + 19\n\ndef step_20(x: int) -> int:\n    \"\"\"Return x plus 20.\"\"\"\n    return x + 20\n\ndef step_21(x: int) -> int:\n    \"\"\"Return x plus 21.\"\"\"\n    return x + 21\n\ndef step_22(x: int) -> int:\n    \"\"\"Return x plus 22.\"\"\"\n    return x + 22\n\ndef step_23(x: int) -> int:\n    \"\"\"Return x plus 23.\"\"\"\n    return x + 23\n\ndef step_24(x: int) -> int:\n    \"\"\"Return x plus 24.\"\"\"\n    return x + 24\n\ndef step_25(x: int) -> int:\n    \"\"\"Return x plus 25.\"\"\"\n    return x + 25\n\ndef step_26(x: int) -> int:\n    \"\"\"Return x plus 26.\"\"\"\n    return x + 26\n\ndef step_27(x: int) -> int:\n    \"\"\"Return x plus 27.\"\"\"\n    return x + 27\n\ndef step_28(x: int) -> int:\n    \"\"\"Return x plus 28.\"\"\"\n    return x + 28\n\ndef step_29(x: int) -> int:\n    \"\"\"Return x plus 29.\"\"\"\n    return x + 29\n\ndef step_30(x: int) -> int:\n    \"\"\"Return x plus 30.\"\"\"\n    return x + 30\n\ndef step_31(x: int) -> int:\n    \"\"\"Return x plus 31.\"\"\"\n    return x + 31\n\ndef step_32(x: int) -> int:\n    \"\"\"Return x plus 32.\"\"\"\n    return x + 32\n\ndef step_33(x: int) -> int:\n    \"\"\"Return x plus 33.\"\"\"\n    return x + 33\n\ndef step_34(x: int) -> int:\n    \"\"\"Return x plus 34.\"\"\"\n    return x + 34\n\ndef step_35(x: int) -> int:\n    \"\"\"Return x plus 35.\"\"\"\n    return x + 35\n\ndef step_36(x: int) -> int:\n    \"\"\"Return x plus 36.\"\"\"\n    return x + 36\n\ndef step_37(x: int) -> int:\n    \"\"\"Return x plus 37.\"\"\"\n    return x + 37\n\ndef step_38(x: int) -> int:\n    \"\"\"Return x plus 38.\"\"\"\n    return x + 38\n\ndef step_39(x: int) -> int:\n    \"\"\"Return x plus 39.\"\"\"\n    return x + 39\n\ndef step_40(x: int) -> int:\n    \"\"\"Return x plus 40.\"\"\"\n    return x + 40\n\ndef step_41(x: int) -> int:\n    \"\"\"Return x plus 41.\"\"\"\n    return x + 41\n\ndef step_42(x: int) -> int:\n    \"\"\"Return x plus 42.\"\"\"\n    return x + 42\n\ndef step_43(x: int) -> int:\n    \"\"\"Return x plus 43.\"\"\"\n    return x + 43\n\ndef step_44(x: int) -> int:\n    \"\"\"Return x plus 44.\"\"\"\n    return x + 44\n\ndef step_45(x: int) -> int:\n    \"\"\"Return x plus 45.\"\"\"\n    return x + 45\n\ndef step_46(x: int) -> int:\n    \"\"\"Return x plus 46.\"\"\"\n    return x + 46\n\ndef step_47(x: int) -> int:\n    \"\"\"Return x plus 47.\"\"\"\n    return x + 47\n\ndef step_48(x: int) -> int:\n    \"\"\"Return x plus 48.\"\"\"\n    return x + 48\n\ndef step_49(x: int) -> int:\n    \"\"\"Return x plus 49.\"\"\"\n    return x + 49\n\ndef step_50(x: int) -> int:\n    \"\"\"Return x plus 50.\"\"\"\n    return x + 50\n\ndef step_51(x: int) -> int:\n    \"\"\"Return x plus 51.\"\"\"\n    return x + 51\n\ndef step_52(x: int) -> int:\n    \"\"\"Return x plus 52.\"\"\"\n    return x + 52\n\ndef step_53(x: int) -> int:\n    \"\"\"Return x plus 53.\"\"\"\n    return x + 53\n\ndef step_54(x: int) -> int:\n    \"\"\"Return x plus 54.\"\"\"\n    return x + 54\n\ndef step_55(x: int) -> int:\n    \"\"\"Return x plus 55.\"\"\"\n    return x + 55\n\ndef step_56(x: int) -> int:\n    \"\"\"Return x plus 56.\"\"\"\n    return x + 56\n\ndef step_57(x: int) -> int:\n    \"\"\"Return x plus 57.\"\"\"\n    return x + 57\n\ndef step_58(x: int) -> int:\n    \"\"\"Return x plus 58.\"\"\"\n    return x + 58\n\ndef step_59(x: int) -> int:\n    \"\"\"Return x plus 59.\"\"\"\n    return x + 59\n",
        "path": "steps.py"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-long-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Write the module.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"steps.py\", \"content\": \"\\\"\\\"\\\"Generated module used as a long tool argument.\\\"\\\"\\\"\\n\\nfrom __future__ import annotations\\n\\ndef step_00(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 0.\\\"\\\"\\\"\\n    return x + 0\\n\\ndef step_01(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 1.\\\"\\\"\\\"\\n    return x + 1\\n\\ndef step_02(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 2.\\\"\\\"\\\"\\n    return x + 2\\n\\ndef step_03(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 3.\\\"\\\"\\\"\\n    return x + 3\\n\\ndef step_04(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 4.\\\"\\\"\\\"\\n    return x + 4\\n\\ndef step_05(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 5.\\\"\\\"\\\"\\n    return x + 5\\n\\ndef step_06(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 6.\\\"\\\"\\\"\\n    return x + 6\\n\\ndef step_07(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 7.\\\"\\\"\\\"\\n    return x + 7\\n\\ndef step_08(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 8.\\\"\\\"\\\"\\n    return x + 8\\n\\ndef step_09(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 9.\\\"\\\"\\\"\\n    return x + 9\\n\\ndef step_10(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 10.\\\"\\\"\\\"\\n    return x + 10\\n\\ndef step_11(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 11.\\\"\\\"\\\"\\n    return x + 11\\n\\ndef step_12(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 12.\\\"\\\"\\\"\\n    return x + 12\\n\\ndef step_13(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 13.\\\"\\\"\\\"\\n    return x + 13\\n\\ndef step_14(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 14.\\\"\\\"\\\"\\n    return x + 14\\n\\ndef step_15(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 15.\\\"\\\"\\\"\\n    return x + 15\\n\\ndef step_16(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 16.\\\"\\\"\\\"\\n    return x + 16\\n\\ndef step_17(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 17.\\\"\\\"\\\"\\n    return x + 17\\n\\ndef step_18(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 18.\\\"\\\"\\\"\\n    return x + 18\\n\\ndef step_19(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 19.\\\"\\\"\\\"\\n    return x + 19\\n\\ndef step_20(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 20.\\\"\\\"\\\"\\n    return x + 20\\n\\ndef step_21(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 21.\\\"\\\"\\\"\\n    return x + 21\\n\\ndef step_22(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 22.\\\"\\\"\\\"\\n    return x + 22\\n\\ndef step_23(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 23.\\\"\\\"\\\"\\n    return x + 23\\n\\ndef step_24(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 24.\\\"\\\"\\\"\\n    return x + 24\\n\\ndef step_25(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 25.\\\"\\\"\\\"\\n    return x + 25\\n\\ndef step_26(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 26.\\\"\\\"\\\"\\n    return x + 26\\n\\ndef step_27(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 27.\\\"\\\"\\\"\\n    return x + 27\\n\\ndef step_28(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 28.\\\"\\\"\\\"\\n    return x + 28\\n\\ndef step_29(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 29.\\\"\\\"\\\"\\n    return x + 29\\n\\ndef step_30(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 30.\\\"\\\"\\\"\\n    return x + 30\\n\\ndef step_31(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 31.\\\"\\\"\\\"\\n    return x + 31\\n\\ndef step_32(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 32.\\\"\\\"\\\"\\n    return x + 32\\n\\ndef step_33(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 33.\\\"\\\"\\\"\\n    return x + 33\\n\\ndef step_34(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 34.\\\"\\\"\\\"\\n    return x + 34\\n\\ndef step_35(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 35.\\\"\\\"\\\"\\n    return x + 35\\n\\ndef step_36(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 36.\\\"\\\"\\\"\\n    return x + 36\\n\\ndef step_37(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 37.\\\"\\\"\\\"\\n    return x + 37\\n\\ndef step_38(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 38.\\\"\\\"\\\"\\n    return x + 38\\n\\ndef step_39(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 39.\\\"\\\"\\\"\\n    return x + 39\\n\\ndef step_40(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 40.\\\"\\\"\\\"\\n    return x + 40\\n\\ndef step_41(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 41.\\\"\\\"\\\"\\n    return x + 41\\n\\ndef step_42(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 42.\\\"\\\"\\\"\\n    return x + 42\\n\\ndef step_43(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 43.\\\"\\\"\\\"\\n    return x + 43\\n\\ndef step_44(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 44.\\\"\\\"\\\"\\n    return x + 44\\n\\ndef step_45(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 45.\\\"\\\"\\\"\\n    return x + 45\\n\\ndef step_46(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 46.\\\"\\\"\\\"\\n    return x + 46\\n\\ndef step_47(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 47.\\\"\\\"\\\"\\n    return x + 47\\n\\ndef step_48(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 48.\\\"\\\"\\\"\\n    return x + 48\\n\\ndef step_49(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 49.\\\"\\\"\\\"\\n    return x + 49\\n\\ndef step_50(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 50.\\\"\\\"\\\"\\n    return x + 50\\n\\ndef step_51(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 51.\\\"\\\"\\\"\\n    return x + 51\\n\\ndef step_52(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 52.\\\"\\\"\\\"\\n    return x + 52\\n\\ndef step_53(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 53.\\\"\\\"\\\"\\n    return x + 53\\n\\ndef step_54(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 54.\\\"\\\"\\\"\\n    return x + 54\\n\\ndef step_55(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 55.\\\"\\\"\\\"\\n    return x + 55\\n\\ndef step_56(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 56.\\\"\\\"\\\"\\n    return x + 56\\n\\ndef step_57(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 57.\\\"\\\"\\\"\\n    return x + 57\\n\\ndef step_58(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 58.\\\"\\\"\\\"\\n    return x + 58\\n\\ndef step_59(x: int) -> int:\\n    \\\"\\\"\\\"Return x plus 59.\\\"\\\"\\\"\\n    return x + 59\\n\"}", "output_token_ids": [200005, 35644, 200008, 10930, 290, 8827, 13, 200007, 200006, 173781, 316, 28, 44580, 6765, 5933, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 4189, 1243, 392, 27176, 17311, 672, 392, 3252, 1243, 32945, 4017, 4017, 24486, 8827, 2061, 472, 261, 1701, 4584, 10383, 186907, 4017, 49819, 77, 3392, 2845, 2516, 48758, 771, 1588, 66872, 3392, 59, 11749, 5983, 62, 504, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 15, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 15, 3392, 59, 11749, 5983, 62, 2290, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 16, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 16, 3392, 59, 11749, 5983, 62, 3286, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 17, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 17, 3392, 59, 11749, 5983, 62, 3659, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 18, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 18, 3392, 59, 11749, 5983, 62, 3000, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 19, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 19, 3392, 59, 11749, 5983, 62, 2922, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 20, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 20, 3392, 59, 11749, 5983, 62, 3218, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 21, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 21, 3392, 59, 11749, 5983, 62, 3173, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 22, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 22, 3392, 59, 11749, 5983, 62, 3062, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 23, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 23, 3392, 59, 11749, 5983, 62, 3114, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 24, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 24, 3392, 59, 11749, 5983, 62, 702, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 702, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 702, 3392, 59, 11749, 5983, 62, 994, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 994, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 994, 3392, 59, 11749, 5983, 62, 899, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 899, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 899, 3392, 59, 11749, 5983, 62, 1311, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1311, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1311, 3392, 59, 11749, 5983, 62, 1265, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1265, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1265, 3392, 59, 11749, 5983, 62, 1055, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1055, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1055, 3392, 59, 11749, 5983, 62, 1125, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1125, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1125, 3392, 59, 11749, 5983, 62, 1422, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1422, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1422, 3392, 59, 11749, 5983, 62, 1157, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1157, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1157, 3392, 59, 11749, 5983, 62, 858, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 858, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 858, 3392, 59, 11749, 5983, 62, 455, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 455, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 455, 3392, 59, 11749, 5983, 62, 2040, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2040, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2040, 3392, 59, 11749, 5983, 62, 1709, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1709, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1709, 3392, 59, 11749, 5983, 62, 1860, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1860, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1860, 3392, 59, 11749, 5983, 62, 1494, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1494, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1494, 3392, 59, 11749, 5983, 62, 1161, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1161, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1161, 3392, 59, 11749, 5983, 62, 2109, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2109, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2109, 3392, 59, 11749, 5983, 62, 2092, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2092, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2092, 3392, 59, 11749, 5983, 62, 2029, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2029, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2029, 3392, 59, 11749, 5983, 62, 2270, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2270, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2270, 3392, 59, 11749, 5983, 62, 1130, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1130, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1130, 3392, 59, 11749, 5983, 62, 2911, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2911, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2911, 3392, 59, 11749, 5983, 62, 1398, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1398, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1398, 3392, 59, 11749, 5983, 62, 2546, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2546, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2546, 3392, 59, 11749, 5983, 62, 3020, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3020, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3020, 3392, 59, 11749, 5983, 62, 2467, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2467, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2467, 3392, 59, 11749, 5983, 62, 2636, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2636, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2636, 3392, 59, 11749, 5983, 62, 2991, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2991, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2991, 3392, 59, 11749, 5983, 62, 3150, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3150, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3150, 3392, 59, 11749, 5983, 62, 3255, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3255, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3255, 3392, 59, 11749, 5983, 62, 1723, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1723, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1723, 3392, 59, 11749, 5983, 62, 4987, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4987, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4987, 3392, 59, 11749, 5983, 62, 4689, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4689, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4689, 3392, 59, 11749, 5983, 62, 5320, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 5320, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 5320, 3392, 59, 11749, 5983, 62, 3336, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3336, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3336, 3392, 59, 11749, 5983, 62, 2548, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 2548, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 2548, 3392, 59, 11749, 5983, 62, 4217, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4217, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4217, 3392, 59, 11749, 5983, 62, 4146, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4146, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4146, 3392, 59, 11749, 5983, 62, 3519, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3519, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3519, 3392, 59, 11749, 5983, 62, 3796, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3796, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3796, 3392, 59, 11749, 5983, 62, 1434, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 1434, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 1434, 3392, 59, 11749, 5983, 62, 6231, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 6231, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 6231, 3392, 59, 11749, 5983, 62, 6283, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 6283, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 6283, 3392, 59, 11749, 5983, 62, 6798, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 6798, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 6798, 3392, 59, 11749, 5983, 62, 6733, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 6733, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 6733, 3392, 59, 11749, 5983, 62, 3152, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 3152, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 3152, 3392, 59, 11749, 5983, 62, 5007, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 5007, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 5007, 3392, 59, 11749, 5983, 62, 5085, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 5085, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 5085, 3392, 59, 11749, 5983, 62, 4635, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4635, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4635, 3392, 59, 11749, 5983, 62, 4621, 4061, 25, 677, 8, 2747, 677, 16008, 77, 271, 14927, 4017, 4017, 8191, 1215, 2932, 220, 4621, 186907, 4017, 49819, 77, 271, 622, 1215, 659, 220, 4621, 3392, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Write the module.", "tool_calls": [{"name": "write_file", "arguments": {"path": "steps.py", "content": "\"\"\"Generated module used as a long tool argument.\"\"\"\n\nfrom __future__ import annotations\n\ndef step_00(x: int) -> int:\n    \"\"\"Return x plus 0.\"\"\"\n    return x + 0\n\ndef step_01(x: int) -> int:\n    \"\"\"Return x plus 1.\"\"\"\n    return x + 1\n\ndef step_02(x: int) -> int:\n    \"\"\"Return x plus 2.\"\"\"\n    return x + 2\n\ndef step_03(x: int) -> int:\n    \"\"\"Return x plus 3.\"\"\"\n    return x + 3\n\ndef step_04(x: int) -> int:\n    \"\"\"Return x plus 4.\"\"\"\n    return x + 4\n\ndef step_05(x: int) -> int:\n    \"\"\"Return x plus 5.\"\"\"\n    return x + 5\n\ndef step_06(x: int) -> int:\n    \"\"\"Return x plus 6.\"\"\"\n    return x + 6\n\ndef step_07(x: int) -> int:\n    \"\"\"Return x plus 7.\"\"\"\n    return x + 7\n\ndef step_08(x: int) -> int:\n    \"\"\"Return x plus 8.\"\"\"\n    return x + 8\n\ndef step_09(x: int) -> int:\n    \"\"\"Return x plus 9.\"\"\"\n    return x + 9\n\ndef step_10(x: int) -> int:\n    \"\"\"Return x plus 10.\"\"\"\n    return x + 10\n\ndef step_11(x: int) -> int:\n    \"\"\"Return x plus 11.\"\"\"\n    return x + 11\n\ndef step_12(x: int) -> int:\n    \"\"\"Return x plus 12.\"\"\"\n    return x + 12\n\ndef step_13(x: int) -> int:\n    \"\"\"Return x plus 13.\"\"\"\n    return x + 13\n\ndef step_14(x: int) -> int:\n    \"\"\"Return x plus 14.\"\"\"\n    return x + 14\n\ndef step_15(x: int) -> int:\n    \"\"\"Return x plus 15.\"\"\"\n    return x + 15\n\ndef step_16(x: int) -> int:\n    \"\"\"Return x plus 16.\"\"\"\n    return x + 16\n\ndef step_17(x: int) -> int:\n    \"\"\"Return x plus 17.\"\"\"\n    return x + 17\n\ndef step_18(x: int) -> int:\n    \"\"\"Return x plus 18.\"\"\"\n    return x + 18\n\ndef step_19(x: int) -> int:\n    \"\"\"Return x plus 19.\"\"\"\n    return x + 19\n\ndef step_20(x: int) -> int:\n    \"\"\"Return x plus 20.\"\"\"\n    return x + 20\n\ndef step_21(x: int) -> int:\n    \"\"\"Return x plus 21.\"\"\"\n    return x + 21\n\ndef step_22(x: int) -> int:\n    \"\"\"Return x plus 22.\"\"\"\n    return x + 22\n\ndef step_23(x: int) -> int:\n    \"\"\"Return x plus 23.\"\"\"\n    return x + 23\n\ndef step_24(x: int) -> int:\n    \"\"\"Return x plus 24.\"\"\"\n    return x + 24\n\ndef step_25(x: int) -> int:\n    \"\"\"Return x plus 25.\"\"\"\n    return x + 25\n\ndef step_26(x: int) -> int:\n    \"\"\"Return x plus 26.\"\"\"\n    return x + 26\n\ndef step_27(x: int) -> int:\n    \"\"\"Return x plus 27.\"\"\"\n    return x + 27\n\ndef step_28(x: int) -> int:\n    \"\"\"Return x plus 28.\"\"\"\n    return x + 28\n\ndef step_29(x: int) -> int:\n    \"\"\"Return x plus 29.\"\"\"\n    return x + 29\n\ndef step_30(x: int) -> int:\n    \"\"\"Return x plus 30.\"\"\"\n    return x + 30\n\ndef step_31(x: int) -> int:\n    \"\"\"Return x plus 31.\"\"\"\n    return x + 31\n\ndef step_32(x: int) -> int:\n    \"\"\"Return x plus 32.\"\"\"\n    return x + 32\n\ndef step_33(x: int) -> int:\n    \"\"\"Return x plus 33.\"\"\"\n    return x + 33\n\ndef step_34(x: int) -> int:\n    \"\"\"Return x plus 34.\"\"\"\n    return x + 34\n\ndef step_35(x: int) -> int:\n    \"\"\"Return x plus 35.\"\"\"\n    return x + 35\n\ndef step_36(x: int) -> int:\n    \"\"\"Return x plus 36.\"\"\"\n    return x + 36\n\ndef step_37(x: int) -> int:\n    \"\"\"Return x plus 37.\"\"\"\n    return x + 37\n\ndef step_38(x: int) -> int:\n    \"\"\"Return x plus 38.\"\"\"\n    return x + 38\n\ndef step_39(x: int) -> int:\n    \"\"\"Return x plus 39.\"\"\"\n    return x + 39\n\ndef step_40(x: int) -> int:\n    \"\"\"Return x plus 40.\"\"\"\n    return x + 40\n\ndef step_41(x: int) -> int:\n    \"\"\"Return x plus 41.\"\"\"\n    return x + 41\n\ndef step_42(x: int) -> int:\n    \"\"\"Return x plus 42.\"\"\"\n    return x + 42\n\ndef step_43(x: int) -> int:\n    \"\"\"Return x plus 43.\"\"\"\n    return x + 43\n\ndef step_44(x: int) -> int:\n    \"\"\"Return x plus 44.\"\"\"\n    return x + 44\n\ndef step_45(x: int) -> int:\n    \"\"\"Return x plus 45.\"\"\"\n    return x + 45\n\ndef step_46(x: int) -> int:\n    \"\"\"Return x plus 46.\"\"\"\n    return x + 46\n\ndef step_47(x: int) -> int:\n    \"\"\"Return x plus 47.\"\"\"\n    return x + 47\n\ndef step_48(x: int) -> int:\n    \"\"\"Return x plus 48.\"\"\"\n    return x + 48\n\ndef step_49(x: int) -> int:\n    \"\"\"Return x plus 49.\"\"\"\n    return x + 49\n\ndef step_50(x: int) -> int:\n    \"\"\"Return x plus 50.\"\"\"\n    return x + 50\n\ndef step_51(x: int) -> int:\n    \"\"\"Return x plus 51.\"\"\"\n    return x + 51\n\ndef step_52(x: int) -> int:\n    \"\"\"Return x plus 52.\"\"\"\n    return x + 52\n\ndef step_53(x: int) -> int:\n    \"\"\"Return x plus 53.\"\"\"\n    return x + 53\n\ndef step_54(x: int) -> int:\n    \"\"\"Return x plus 54.\"\"\"\n    return x + 54\n\ndef step_55(x: int) -> int:\n    \"\"\"Return x plus 55.\"\"\"\n    return x + 55\n\ndef step_56(x: int) -> int:\n    \"\"\"Return x plus 56.\"\"\"\n    return x + 56\n\ndef step_57(x: int) -> int:\n    \"\"\"Return x plus 57.\"\"\"\n    return x + 57\n\ndef step_58(x: int) -> int:\n    \"\"\"Return x plus 58.\"\"\"\n    return x + 58\n\ndef step_59(x: int) -> int:\n    \"\"\"Return x plus 59.\"\"\"\n    return x + 59\n"}}]}, "tags": ["single-call", "reasoning", "long-arguments", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-marker-in-arguments expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-marker-in-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ...; tool_calls: expected ['write_file'], got []
`expected_match`	`one`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`special`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`token`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got '; a turn with .\\nHeader: {}"}'; tool_calls: expected ['write_file'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "harmony.md", "content": "A call ends with <\|call\|>; a ..., got '; a turn with .\\nHeader: {}"}'
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-marker-in-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"harmony.md\", \"content\": \"A call ends with <|call|>; a turn with .\\nHeader: {}\"}",
   "reasoning_content": "Write the Harmony cheat sheet to a file.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
-        "path": "harmony.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "; a turn with .\\nHeader: {}\"}",
   "reasoning_content": "Write the Harmony cheat sheet to a file.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
-        "path": "harmony.md"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Write the Harmony cheat sheet to a file.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{"path": "harmony.md", "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>"}

Expected parse

{
  "content": null,
  "reasoning_content": "Write the Harmony cheat sheet to a file.",
  "tool_calls": [
    {
      "arguments": {
        "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>",
        "path": "harmony.md"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-marker-in-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Write the Harmony cheat sheet to a file.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"harmony.md\", \"content\": \"A call ends with <|call|>; a turn with <|return|>.\\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>\"}", "output_token_ids": [200005, 35644, 200008, 10930, 290, 102323, 65354, 16658, 316, 261, 1974, 13, 200007, 200006, 173781, 316, 28, 44580, 6765, 5933, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 4189, 1243, 392, 71, 90047, 44190, 672, 392, 3252, 1243, 392, 32, 2421, 17095, 483, 464, 91, 9925, 91, 170897, 261, 3716, 483, 464, 91, 1034, 91, 29, 15043, 77, 5737, 25, 464, 91, 5236, 91, 29, 173781, 27, 91, 21453, 91, 29, 12606, 815, 316, 28, 44580, 775, 6425, 464, 91, 542, 141043, 91, 29, 4108, 27, 91, 3938, 91, 11679, 157249, 91, 419, 91, 29, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Write the Harmony cheat sheet to a file.", "tool_calls": [{"name": "write_file", "arguments": {"path": "harmony.md", "content": "A call ends with <|call|>; a turn with <|return|>.\nHeader: <|start|>assistant<|channel|>commentary to=functions.get_time <|constrain|>json<|message|>{}<|end|>"}}]}, "tags": ["single-call", "reasoning", "marker-in-arguments", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). The special-token strings inside the argument value are ORDINARY text tokens in output_token_ids (openai-harmony encodes message content with special tokens disallowed). A parser that works on text rather than token ids cannot tell them from real markers."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-marker-in-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/harmony-marker-in-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>'; reasoning_content: expected 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.', got None; tool_calls: expected ['get_time'], got []
`expected_match`	`one`	fail	content: expected None, got ' or {}'; reasoning_content: expected 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.', got None; tool_calls: expected ['get_time'], got []
`expected_match`	`special`	fail	content: expected None, got ' or {}'; reasoning_content: expected 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.', got None; tool_calls: expected ['get_time'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got ' or {}'; reasoning_content: expected 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.', got 'I must not write'; tool_calls: expected ['get_time'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_time'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got ' or {}'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got ' or {}'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got None; reasoning_content: nonstream None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got None; reasoning_content: nonstream None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got None; reasoning_content: nonstream None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got ' or {}'; reasoning_content: nonstream None, got 'I must not write'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got None; reasoning_content: nonstream None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|channel\|>analysis<\|message\|>I must not write <\|call\|> or {}<\|call\|>', got None; reasoning_content: nonstream None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`split_invariance`	`*`	fail	vs one: token (fail): content: one ' or {}', got None; reasoning_content: one None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.' \| rand:1:8 (fail): content: one ' or {}', got None; reasoning_content: one None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.' \| rand:2:8 (fail): content: one ' or {}', got None; reasoning_content: one None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.' \| rand:3:8 (fail): reasoning_content: one None, got 'I must not write' \| rand:4:8 (fail): content: one ' or {}', got None; reasoning_content: one None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.' \| rand:5:8 (fail): content: one ' or {}', got None; reasoning_content: one None, got 'I must not write <\|call\|> or <\|channel\|>final myself; the tool call does it.'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-marker-in-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 8 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "content": "<|channel|>analysis<|message|>I must not write <|call|> or {}<|call|>",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: one, special

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "content": " or {}",
+  "reasoning_content": null,
+  "tool_calls": []
 }

Strategies: token, rand:1:8, rand:2:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "content": " or {}",
+  "reasoning_content": "I must not write",
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>I must not write <|call|> or <|channel|>final myself; the tool call does it.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary <|constrain|>json<|message|>{}

Expected parse

{
  "content": null,
  "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-marker-in-reasoning", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>I must not write <|call|> or <|channel|>final myself; the tool call does it.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary <|constrain|>json<|message|>{}", "output_token_ids": [200005, 35644, 200008, 40, 2804, 625, 5067, 464, 91, 9925, 91, 29, 503, 464, 91, 21453, 91, 29, 17196, 9012, 26, 290, 4584, 2421, 2226, 480, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 220, 200003, 4108, 200008, 12083], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "I must not write <|call|> or <|channel|>final myself; the tool call does it.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "reasoning", "empty-arguments", "x-marker-in-reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). Marker strings inside the analysis text are ordinary text tokens."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-multiline-reasoning expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-multiline-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>'; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "weather Paris"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-multiline-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 17 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"weather Paris\"}<|call|>",
   "reasoning_content": "Step 1: parse the request.\nStep 2: it needs live data.\n\nStep 3: call search.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "weather Paris"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "Step 1: parse the request.\nStep 2: it needs live data.\n\nStep 3: call search.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "weather Paris"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, reasoning, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Step 1: parse the request.
Step 2: it needs live data.

Step 3: call search.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{"query": "weather Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": "Step 1: parse the request.\nStep 2: it needs live data.\n\nStep 3: call search.",
  "tool_calls": [
    {
      "arguments": {
        "query": "weather Paris"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-multiline-reasoning", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Step 1: parse the request.\nStep 2: it needs live data.\n\nStep 3: call search.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"weather Paris\"}", "output_token_ids": [200005, 35644, 200008, 10643, 220, 16, 25, 8420, 290, 2616, 558, 10643, 220, 17, 25, 480, 4414, 4561, 1238, 364, 10643, 220, 18, 25, 2421, 3684, 13, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 2975, 1243, 392, 28393, 12650, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Step 1: parse the request.\nStep 2: it needs live data.\n\nStep 3: call search.", "tool_calls": [{"name": "search", "arguments": {"query": "weather Paris"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-nested-json expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-nested-json
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"...; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "café \\"best\\"", "filters": {"tags": ["a", "b"], "range"..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-nested-json --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,30 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"range\": {\"from\": 1, \"to\": 3, \"open\": null}, \"sort\": [{\"field\": \"rating\", \"desc\": true}], \"exact\": false}}<|call|>",
   "reasoning_content": "Search with structured filters.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "exact": false,
-          "range": {
-            "from": 1,
-            "open": null,
-            "to": 3
-          },
-          "sort": [
-            {
-              "desc": true,
-              "field": "rating"
-            }
-          ],
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,29 +2,4 @@
   "content": null,
   "reasoning_content": "Search with structured filters.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "exact": false,
-          "range": {
-            "from": 1,
-            "open": null,
-            "to": 3
-          },
-          "sort": [
-            {
-              "desc": true,
-              "field": "rating"
-            }
-          ],
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\""
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Search with structured filters.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{"query": "café \"best\"", "filters": {"tags": ["a", "b"], "range": {"from": 1, "to": 3, "open": null}, "sort": [{"field": "rating", "desc": true}], "exact": false}}

Expected parse

{
  "content": null,
  "reasoning_content": "Search with structured filters.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "exact": false,
          "range": {
            "from": 1,
            "open": null,
            "to": 3
          },
          "sort": [
            {
              "desc": true,
              "field": "rating"
            }
          ],
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\""
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-nested-json", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Search with structured filters.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"café \\\"best\\\"\", \"filters\": {\"tags\": [\"a\", \"b\"], \"range\": {\"from\": 1, \"to\": 3, \"open\": null}, \"sort\": [{\"field\": \"rating\", \"desc\": true}], \"exact\": false}}", "output_token_ids": [200005, 35644, 200008, 5975, 483, 47557, 21205, 13, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 2975, 1243, 392, 66, 103112, 14927, 17844, 140800, 392, 50376, 1243, 10494, 27989, 1243, 9129, 64, 672, 392, 65, 17695, 392, 20967, 1243, 10494, 2845, 1243, 220, 16, 11, 392, 935, 1243, 220, 18, 11, 392, 3843, 1243, 1256, 4180, 392, 13373, 1243, 161269, 4735, 1243, 392, 42877, 672, 392, 17186, 1243, 1343, 171092, 392, 86898, 1243, 1485, 6478], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Search with structured filters.", "tool_calls": [{"name": "search", "arguments": {"query": "café \"best\"", "filters": {"tags": ["a", "b"], "range": {"from": 1, "to": 3, "open": null}, "sort": [{"field": "rating", "desc": true}], "exact": false}}}]}, "tags": ["single-call", "reasoning", "nested-json", "string-escapes", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-no-content-type expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-no-content-type
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|message\|>{"city": "Paris"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-no-content-type --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 19 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary<|message|>{\"city\": \"Paris\"}<|call|>",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|message|>{"city": "Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-no-content-type", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|message|>{\"city\": \"Paris\"}", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200008, 10848, 17500, 1243, 392, 72782, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-no-content-type"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). Tool call header without any content type, as in llama.cpp's gpt-oss tests."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-numeric-arguments expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-numeric-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"...; tool_calls: expected ['set_alarm'], got []
`expected_match`	`one`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`special`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`token`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['set_alarm'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.set_alarm<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label"..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-numeric-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 15 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,16 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.set_alarm<|channel|>commentary <|constrain|>json<|message|>{\"hour\": 7, \"minute\": 30, \"volume\": 0.75, \"repeat\": true, \"label\": null}<|call|>",
   "reasoning_content": "Set the alarm.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "hour": 7,
-        "label": null,
-        "minute": 30,
-        "repeat": true,
-        "volume": 0.75
-      },
-      "name": "set_alarm"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,15 +2,4 @@
   "content": null,
   "reasoning_content": "Set the alarm.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "hour": 7,
-        "label": null,
-        "minute": 30,
-        "repeat": true,
-        "volume": 0.75
-      },
-      "name": "set_alarm"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Set the alarm.<|end|><|start|>assistant to=functions.set_alarm<|channel|>commentary <|constrain|>json<|message|>{"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label": null}

Expected parse

{
  "content": null,
  "reasoning_content": "Set the alarm.",
  "tool_calls": [
    {
      "arguments": {
        "hour": 7,
        "label": null,
        "minute": 30,
        "repeat": true,
        "volume": 0.75
      },
      "name": "set_alarm"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-numeric-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Set the alarm.<|end|><|start|>assistant to=functions.set_alarm<|channel|>commentary <|constrain|>json<|message|>{\"hour\": 7, \"minute\": 30, \"volume\": 0.75, \"repeat\": true, \"label\": null}", "output_token_ids": [200005, 35644, 200008, 1984, 290, 23465, 13, 200007, 200006, 173781, 316, 28, 44580, 1230, 160145, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 50094, 1243, 220, 22, 11, 392, 84372, 1243, 220, 1130, 11, 392, 23661, 1243, 220, 15, 13, 3384, 11, 392, 63242, 1243, 1343, 11, 392, 3198, 1243, 1256, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Set the alarm.", "tool_calls": [{"name": "set_alarm", "arguments": {"hour": 7, "minute": 30, "volume": 0.75, "repeat": true, "label": null}}]}, "tags": ["single-call", "reasoning", "numeric-arguments", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-preamble-no-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/harmony-preamble-no-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c...; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'; tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Checking two sources — one moment.<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "vLLM release notes"}<\|c..., got 'Checking two sources — one moment.Checking two sources — one moment.'
`split_invariance`	`*`	fail	vs one: special (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| token (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| rand:1:8 (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| rand:2:8 (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| rand:3:8 (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| rand:4:8 (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.' \| rand:5:8 (fail): content: one 'Checking two sources — one moment.', got 'Checking two sources — one moment.Checking two sources — one moment.'
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-preamble-no-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 10 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Checking two sources — one moment.",
+  "content": "Checking two sources — one moment.<|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"vLLM release notes\"}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vLLM release notes"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": "Checking two sources — one moment.",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vLLM release notes"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Checking two sources — one moment.",
+  "content": "Checking two sources — one moment.Checking two sources — one moment.",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vLLM release notes"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, text-before-call, x-preamble-commentary, x-recipient-in-role.

Raw output

<|channel|>commentary<|message|>Checking two sources — one moment.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{"query": "vLLM release notes"}

Expected parse

{
  "content": "Checking two sources — one moment.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "query": "vLLM release notes"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-preamble-no-reasoning", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>commentary<|message|>Checking two sources — one moment.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"vLLM release notes\"}", "output_token_ids": [200005, 12606, 815, 200008, 70142, 1920, 11525, 2733, 1001, 4205, 13, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 2975, 1243, 392, 85, 7454, 44, 8232, 12870, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Checking two sources — one moment.", "reasoning_content": null, "tool_calls": [{"name": "search", "arguments": {"query": "vLLM release notes"}}]}, "tags": ["single-call", "text-before-call", "x-preamble-commentary", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-preamble-then-call expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/harmony-preamble-then-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}...; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.'
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Let me check the weather for you.<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}..., got 'Let me check the weather for you.Let me check the weather for you.'
`split_invariance`	`*`	fail	vs one: special (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| token (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| rand:1:8 (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| rand:2:8 (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| rand:3:8 (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| rand:4:8 (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.' \| rand:5:8 (fail): content: one 'Let me check the weather for you.', got 'Let me check the weather for you.Let me check the weather for you.'
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-preamble-then-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 9 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you.<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}<|call|>",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": "Let me check the weather for you.",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": "Let me check the weather for you.",
+  "content": "Let me check the weather for you.Let me check the weather for you.",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant<|channel|>commentary<|message|>Let me check the weather for you.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris", "unit": "c"}

Expected parse

{
  "content": "Let me check the weather for you.",
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-preamble-then-call", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant<|channel|>commentary<|message|>Let me check the weather for you.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 200005, 12606, 815, 200008, 12845, 668, 2371, 290, 11122, 395, 481, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400, 1243, 392, 66, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Let me check the weather for you.", "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "text-before-call", "x-preamble-commentary", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). A commentary message without a recipient is a user-visible preamble (Harmony spec), so it is content."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-single-call expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 1 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}<|call|>",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris", "unit": "c"}

Expected parse

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-single-call", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400, 1243, 392, 66, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-constrain-token"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-single-call-no-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/harmony-single-call-no-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	content: expected None, got ' to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather'
`stream_equals_nonstream`	`special`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`token`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"city": "Paris"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`split_invariance`	`*`	fail	vs one: special (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| token (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:1:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:2:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:3:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:4:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:5:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-single-call-no-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 2 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\"}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather to=functions.get_weather",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, x-recipient-in-role, x-constrain-token.

Raw output

 to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{"city": "Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-single-call-no-reasoning", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": " to=functions.get_weather<|channel|>commentary <|constrain|>json<|message|>{\"city\": \"Paris\"}", "output_token_ids": [316, 28, 44580, 775, 170154, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 17500, 1243, 392, 72782, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris"}}]}, "tags": ["single-call", "x-recipient-in-role", "x-constrain-token"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header). The model skipped the analysis channel and started with the tool-call message."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-spec-preamble-action-plan expected_match, stream_equals_nonstream, split_invariance

Checks that did not pass for gpt-oss/harmony-spec-preamble-action-plan
Check	Strategy	Result	Detail
`expected_match`	`special`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`token`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`rand:1:8`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`rand:2:8`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`rand:3:8`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`rand:4:8`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`expected_match`	`rand:5:8`	fail	content: expected 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`special`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`token`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...
`split_invariance`	`*`	fail	vs one: special (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| token (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| rand:1:8 (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| rand:2:8 (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| rand:3:8 (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| rand:4:8 (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by... \| rand:5:8 (fail): content: one 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by..., got 'Action plan:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by...

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/harmony-spec-preamble-action-plan --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 21 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step",
+  "content": "**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step",
   "reasoning_content": "{long chain of thought}",
   "tool_calls": [

Strategies: nonstream, one

Matches the expected parse.

{
  "content": "**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step",
  "reasoning_content": "{long chain of thought}",
  "tool_calls": [
    {
      "arguments": {
        "path": "index.html",
        "template": "basic_html"
      },
      "name": "generate_file"
    }
  ]
}

Fixture

Provenance: spec_example, https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/docs/format.md#L468-L474. Tags: single-call, reasoning, text-before-call, x-preamble-commentary, x-recipient-in-channel, x-constrain-nospace.

Raw output

<|channel|>analysis<|message|>{long chain of thought}<|end|><|start|>assistant<|channel|>commentary<|message|>**Action plan**:
1. Generate an HTML file
2. Generate a JavaScript for the Node.js server
3. Start the server
---
Will start executing the plan step by step<|end|><|start|>assistant<|channel|>commentary to=functions.generate_file<|constrain|>json<|message|>{"template": "basic_html", "path": "index.html"}

Expected parse

{
  "content": "**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step",
  "reasoning_content": "{long chain of thought}",
  "tool_calls": [
    {
      "arguments": {
        "path": "index.html",
        "template": "basic_html"
      },
      "name": "generate_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-spec-preamble-action-plan", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "spec_example", "source_url": "https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/docs/format.md#L468-L474", "revision": "abd677f7ac962629c808197caa1feb9e3e95d2b0", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright OpenAI (openai/harmony, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "generate_file", "parameters": {"type": "object", "properties": {"template": {"type": "string"}, "path": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>{long chain of thought}<|end|><|start|>assistant<|channel|>commentary<|message|>**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step<|end|><|start|>assistant<|channel|>commentary to=functions.generate_file<|constrain|>json<|message|>{\"template\": \"basic_html\", \"path\": \"index.html\"}", "output_token_ids": [200005, 35644, 200008, 90, 8431, 13464, 328, 4525, 92, 200007, 200006, 173781, 200005, 12606, 815, 200008, 410, 3541, 3496, 410, 734, 16, 13, 33886, 448, 15961, 1974, 198, 17, 13, 33886, 261, 13114, 9991, 395, 290, 10882, 5391, 6017, 198, 18, 13, 7972, 290, 6017, 198, 58189, 17886, 1604, 58913, 290, 3496, 5983, 656, 5983, 200007, 200006, 173781, 200005, 12606, 815, 316, 28, 44580, 33917, 5933, 200003, 4108, 200008, 10848, 8314, 1243, 392, 45235, 20821, 672, 392, 4189, 1243, 392, 2257, 4588, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "**Action plan**:\n1. Generate an HTML file\n2. Generate a JavaScript for the Node.js server\n3. Start the server\n---\nWill start executing the plan step by step", "reasoning_content": "{long chain of thought}", "tool_calls": [{"name": "generate_file", "arguments": {"template": "basic_html", "path": "index.html"}}]}, "tags": ["single-call", "reasoning", "text-before-call", "x-preamble-commentary", "x-recipient-in-channel", "x-constrain-nospace"], "notes": "The Harmony spec's preamble example (section 'Preambles'). '{long chain of thought}' is the spec's literal placeholder text. The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/harmony-string-escapes expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-string-escapes
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n...; tool_calls: expected ['write_file'], got []
`expected_match`	`one`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`special`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`token`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['write_file'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['write_file'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.write_file<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"path": "C:\\\\Users\\\\me\\\\notes.txt", "content": "line 1\\n..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-string-escapes --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"C:\\\\Users\\\\me\\\\notes.txt\", \"content\": \"line 1\\n\\t\\\"quoted\\\" and \\\\backslash\\\\\\r\\nend \\u0007\"}<|call|>",
   "reasoning_content": "Write a Windows path and a quoted line.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "line 1\n\t\"quoted\" and \\backslash\\\r\nend \u0007",
-        "path": "C:\\Users\\me\\notes.txt"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": "Write a Windows path and a quoted line.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "content": "line 1\n\t\"quoted\" and \\backslash\\\r\nend \u0007",
-        "path": "C:\\Users\\me\\notes.txt"
-      },
-      "name": "write_file"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Write a Windows path and a quoted line.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{"path": "C:\\Users\\me\\notes.txt", "content": "line 1\n\t\"quoted\" and \\backslash\\\r\nend \u0007"}

Expected parse

{
  "content": null,
  "reasoning_content": "Write a Windows path and a quoted line.",
  "tool_calls": [
    {
      "arguments": {
        "content": "line 1\n\t\"quoted\" and \\backslash\\\r\nend \u0007",
        "path": "C:\\Users\\me\\notes.txt"
      },
      "name": "write_file"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-string-escapes", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Write a Windows path and a quoted line.<|end|><|start|>assistant to=functions.write_file<|channel|>commentary <|constrain|>json<|message|>{\"path\": \"C:\\\\Users\\\\me\\\\notes.txt\", \"content\": \"line 1\\n\\t\\\"quoted\\\" and \\\\backslash\\\\\\r\\nend \\u0007\"}", "output_token_ids": [200005, 35644, 200008, 10930, 261, 7206, 3104, 326, 261, 43531, 2543, 13, 200007, 200006, 173781, 316, 28, 44580, 6765, 5933, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 4189, 1243, 392, 34, 47754, 10554, 5994, 1047, 5994, 38705, 7186, 672, 392, 3252, 1243, 392, 1137, 220, 16, 3392, 10229, 4017, 121546, 4017, 326, 56557, 2205, 117501, 5994, 59, 81, 3392, 419, 2381, 84, 1302, 22, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Write a Windows path and a quoted line.", "tool_calls": [{"name": "write_file", "arguments": {"path": "C:\\Users\\me\\notes.txt", "content": "line 1\n\t\"quoted\" and \\backslash\\\r\nend \u0007"}}]}, "tags": ["single-call", "reasoning", "string-escapes", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/harmony-test-channel-first-constrain-adjacent expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-test-channel-first-constrain-adjacent
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.get_weather<\|constrain\|>json<\|message\|>{"latitude":48.8566,"longitude":2.3522}<\|call\|>', got None; tool_calls: nonstream [], got ['get_weather']
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/harmony-test-channel-first-constrain-adjacent --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 18 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|channel|>commentary to=functions.get_weather<|constrain|>json<|message|>{\"latitude\":48.8566,\"longitude\":2.3522}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "latitude": 48.8566,
-        "longitude": 2.3522
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Matches the expected parse.

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "latitude": 48.8566,
        "longitude": 2.3522
      },
      "name": "get_weather"
    }
  ]
}

Fixture

Provenance: engine_test, https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/tests/test_harmony.py#L300-L322. Tags: single-call, numeric-arguments, x-recipient-in-channel, x-constrain-nospace.

Raw output

<|channel|>commentary to=functions.get_weather<|constrain|>json<|message|>{"latitude":48.8566,"longitude":2.3522}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "latitude": 48.8566,
        "longitude": 2.3522
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-test-channel-first-constrain-adjacent", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/tests/test_harmony.py#L300-L322", "revision": "abd677f7ac962629c808197caa1feb9e3e95d2b0", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright OpenAI (openai/harmony, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"latitude": {"type": "number"}, "longitude": {"type": "number"}}}}}], "raw_output": "<|channel|>commentary to=functions.get_weather<|constrain|>json<|message|>{\"latitude\":48.8566,\"longitude\":2.3522}", "output_token_ids": [200005, 12606, 815, 316, 28, 44580, 775, 170154, 200003, 4108, 200008, 10848, 56795, 1243, 3519, 13, 48439, 21, 3532, 60036, 1243, 17, 13, 31083, 17, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"latitude": 48.8566, "longitude": 2.3522}}]}, "tags": ["single-call", "numeric-arguments", "x-recipient-in-channel", "x-constrain-nospace"], "notes": "test_tool_call_with_channel_before_recipient_and_constrain_adjacent. The source's leading <|start|>assistant is the generation prompt and was removed. The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/harmony-test-role-recipient-constrain-adjacent expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/harmony-test-role-recipient-constrain-adjacent
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	content: expected None, got ' to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got ' to=functions.get_weather to=functions.get_weather'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather'
`stream_equals_nonstream`	`special`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`token`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream ' to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Tokyo"}<\|call\|>', got ' to=functions.get_weather to=functions.get_weather'
`split_invariance`	`*`	fail	vs one: special (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| token (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:1:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:2:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:3:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:4:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather' \| rand:5:8 (fail): content: one ' to=functions.get_weather', got ' to=functions.get_weather to=functions.get_weather'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/harmony-test-role-recipient-constrain-adjacent --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 19 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Tokyo\"}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.get_weather to=functions.get_weather",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Tokyo"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/tests/test_harmony.py#L272-L292. Tags: single-call, x-recipient-in-role, x-constrain-nospace.

Raw output

 to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{"location": "Tokyo"}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "Tokyo"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-test-role-recipient-constrain-adjacent", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/openai/harmony/blob/abd677f7ac962629c808197caa1feb9e3e95d2b0/tests/test_harmony.py#L272-L292", "revision": "abd677f7ac962629c808197caa1feb9e3e95d2b0", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright OpenAI (openai/harmony, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": " to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Tokyo\"}", "output_token_ids": [316, 28, 44580, 775, 170154, 200005, 12606, 815, 200003, 4108, 200008, 10848, 7693, 1243, 392, 173844, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "Tokyo"}}]}, "tags": ["single-call", "x-recipient-in-role", "x-constrain-nospace"], "notes": "test_tool_call_with_constrain_marker_adjacent: 'the model might not output a space before constrain'. The source's leading <|start|>assistant is the generation prompt and was removed. The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/harmony-unicode-emoji expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/harmony-unicode-emoji
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"...; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary <\|constrain\|>json<\|message\|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang"..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/harmony-unicode-emoji --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,15 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"東京の天気 ☀️🌧️ — «prévisions» für Zürich\", \"filters\": {\"lang\": \"ja\"}}<|call|>",
   "reasoning_content": "用户想知道东京的天气。Call the tool 🌦️.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "lang": "ja"
-        },
-        "query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,14 +2,4 @@
   "content": null,
   "reasoning_content": "用户想知道东京的天气。Call the tool 🌦️.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "lang": "ja"
-        },
-        "query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs. Tags: single-call, reasoning, unicode, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>用户想知道东京的天气。Call the tool 🌦️.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang": "ja"}}

Expected parse

{
  "content": null,
  "reasoning_content": "用户想知道东京的天气。Call the tool 🌦️.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "lang": "ja"
        },
        "query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/harmony-unicode-emoji", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://github.com/openai/harmony/blob/ec7606df9e87e3d0a1fec9f50928c1e407f0c438/src/encoding.rs", "revision": "ec7606df9e87e3d0a1fec9f50928c1e407f0c438", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "788675d601fe782e0c9aa7b1111166498a3ab380ca885b7d164e27f7438184ed", "attribution": "Rendered with openai-harmony 0.0.8 (Copyright OpenAI, Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>用户想知道东京的天气。Call the tool 🌦️.<|end|><|start|>assistant to=functions.search<|channel|>commentary <|constrain|>json<|message|>{\"query\": \"東京の天気 ☀️🌧️ — «prévisions» für Zürich\", \"filters\": {\"lang\": \"ja\"}}", "output_token_ids": [200005, 35644, 200008, 24367, 18165, 51000, 103063, 1616, 167823, 788, 4701, 290, 4584, 130321, 99, 15148, 13, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 220, 200003, 4108, 200008, 10848, 2975, 1243, 392, 108713, 3385, 867, 25717, 25701, 222, 15148, 64364, 100, 15148, 2733, 2415, 192449, 26422, 1924, 3125, 127483, 672, 392, 50376, 1243, 10494, 7585, 1243, 392, 2067, 57612], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "用户想知道东京的天气。Call the tool 🌦️.", "tool_calls": [{"name": "search", "arguments": {"query": "東京の天気 ☀️🌧️ — «prévisions» für Zürich", "filters": {"lang": "ja"}}}]}, "tags": ["single-call", "reasoning", "unicode", "x-recipient-in-role"], "notes": "History render by openai-harmony 0.0.8 (recipient in the role header)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/hf-template-empty-arguments expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/hf-template-empty-arguments
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>'; tool_calls: expected ['get_time'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_time'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_time'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_time<\|channel\|>commentary json<\|message\|>{}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/hf-template-empty-arguments --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 25 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,10 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_time<|channel|>commentary json<|message|>{}<|call|>",
   "reasoning_content": "Get the time.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,9 +2,4 @@
   "content": null,
   "reasoning_content": "Get the time.",
-  "tool_calls": [
-    {
-      "arguments": {},
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja. Tags: single-call, reasoning, empty-arguments, x-recipient-in-role, x-constrain-text.

Raw output

<|channel|>analysis<|message|>Get the time.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary json<|message|>{}

Expected parse

{
  "content": null,
  "reasoning_content": "Get the time.",
  "tool_calls": [
    {
      "arguments": {},
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/hf-template-empty-arguments", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Get the time.<|end|><|start|>assistant to=functions.get_time<|channel|>commentary json<|message|>{}", "output_token_ids": [200005, 35644, 200008, 1816, 290, 1058, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 5701, 200008, 12083], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Get the time.", "tool_calls": [{"name": "get_time", "arguments": {}}]}, "tags": ["single-call", "reasoning", "empty-arguments", "x-recipient-in-role", "x-constrain-text"], "notes": "History render by the HF chat_template.jinja (recipient in the role header, content type written as plain ' json'). The template renders only tool_calls[0] of a message."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/hf-template-no-reasoning expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/hf-template-no-reasoning
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`one`	fail	content: expected None, got ' to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`special`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`token`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got ' to=functions.set_alarm to=functions.set_alarm'; tool_calls: expected ['set_alarm'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm'
`stream_equals_nonstream`	`special`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`token`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream ' to=functions.set_alarm<\|channel\|>commentary json<\|message\|>{"hour": 6, "minute": 45, "repeat": false}<\|call\|>', got ' to=functions.set_alarm to=functions.set_alarm'
`split_invariance`	`*`	fail	vs one: special (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| token (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| rand:1:8 (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| rand:2:8 (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| rand:3:8 (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| rand:4:8 (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm' \| rand:5:8 (fail): content: one ' to=functions.set_alarm', got ' to=functions.set_alarm to=functions.set_alarm'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/hf-template-no-reasoning --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 26 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,14 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.set_alarm<|channel|>commentary json<|message|>{\"hour\": 6, \"minute\": 45, \"repeat\": false}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "hour": 6,
-        "minute": 45,
-        "repeat": false
-      },
-      "name": "set_alarm"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -1,14 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.set_alarm",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "hour": 6,
-        "minute": 45,
-        "repeat": false
-      },
-      "name": "set_alarm"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,14 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.set_alarm to=functions.set_alarm",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "hour": 6,
-        "minute": 45,
-        "repeat": false
-      },
-      "name": "set_alarm"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja. Tags: single-call, numeric-arguments, x-recipient-in-role, x-constrain-text.

Raw output

 to=functions.set_alarm<|channel|>commentary json<|message|>{"hour": 6, "minute": 45, "repeat": false}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "hour": 6,
        "minute": 45,
        "repeat": false
      },
      "name": "set_alarm"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/hf-template-no-reasoning", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": " to=functions.set_alarm<|channel|>commentary json<|message|>{\"hour\": 6, \"minute\": 45, \"repeat\": false}", "output_token_ids": [316, 28, 44580, 1230, 160145, 200005, 12606, 815, 5701, 200008, 10848, 50094, 1243, 220, 21, 11, 392, 84372, 1243, 220, 2548, 11, 392, 63242, 1243, 1485, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "set_alarm", "arguments": {"hour": 6, "minute": 45, "repeat": false}}]}, "tags": ["single-call", "numeric-arguments", "x-recipient-in-role", "x-constrain-text"], "notes": "History render by the HF chat_template.jinja (recipient in the role header, content type written as plain ' json'). The template renders only tool_calls[0] of a message."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/hf-template-single-call expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/hf-template-single-call
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary json<\|message\|>{"city": "Paris", "unit": "c"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/hf-template-single-call --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 23 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,13 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}<|call|>",
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,12 +2,4 @@
   "content": null,
   "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "city": "Paris",
-        "unit": "c"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{"city": "Paris", "unit": "c"}

Expected parse

{
  "content": null,
  "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.",
  "tool_calls": [
    {
      "arguments": {
        "city": "Paris",
        "unit": "c"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/hf-template-single-call", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>The user asks for the weather in Paris. I should call get_weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary json<|message|>{\"city\": \"Paris\", \"unit\": \"c\"}", "output_token_ids": [200005, 35644, 200008, 976, 1825, 31064, 395, 290, 11122, 306, 12650, 13, 357, 1757, 2421, 717, 170154, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 5701, 200008, 10848, 17500, 1243, 392, 72782, 672, 392, 5400, 1243, 392, 66, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "The user asks for the weather in Paris. I should call get_weather.", "tool_calls": [{"name": "get_weather", "arguments": {"city": "Paris", "unit": "c"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-constrain-text"], "notes": "History render by the HF chat_template.jinja (recipient in the role header, content type written as plain ' json'). The template renders only tool_calls[0] of a message."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/hf-template-unicode-nested expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/hf-template-unicode-nested
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<...; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary json<\|message\|>{"query": "café \\"best\\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}<..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/rendered.jsonl --id gpt-oss/hf-template-unicode-nested --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 24 of fixtures/gpt-oss/rendered.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,19 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.search<|channel|>commentary json<|message|>{\"query\": \"café \\\"best\\\" Zürich ☕\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}<|call|>",
   "reasoning_content": "Suche nach dem besten Café in Zürich.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\" Zürich ☕"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,18 +2,4 @@
   "content": null,
   "reasoning_content": "Suche nach dem besten Café in Zürich.",
-  "tool_calls": [
-    {
-      "arguments": {
-        "filters": {
-          "max": 3,
-          "tags": [
-            "a",
-            "b"
-          ]
-        },
-        "query": "café \"best\" Zürich ☕"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: template_render, https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja. Tags: single-call, reasoning, unicode, nested-json, string-escapes, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Suche nach dem besten Café in Zürich.<|end|><|start|>assistant to=functions.search<|channel|>commentary json<|message|>{"query": "café \"best\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}

Expected parse

{
  "content": null,
  "reasoning_content": "Suche nach dem besten Café in Zürich.",
  "tool_calls": [
    {
      "arguments": {
        "filters": {
          "max": 3,
          "tags": [
            "a",
            "b"
          ]
        },
        "query": "café \"best\" Zürich ☕"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/hf-template-unicode-nested", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "template_render", "source_url": "https://huggingface.co/openai/gpt-oss-20b/blob/6cee5e81ee83917806bbde320786a8fb61efebee/chat_template.jinja", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/render_gpt_oss.py", "template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146"}, "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["c", "f"]}}, "required": ["city"]}}}, {"type": "function", "function": {"name": "search", "description": "Search the web.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "object"}}, "required": ["query"]}}}, {"type": "function", "function": {"name": "get_time", "description": "Get the current UTC time.", "parameters": {"type": "object", "properties": {}, "required": []}}}, {"type": "function", "function": {"name": "write_file", "description": "Write text to a file.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}, {"type": "function", "function": {"name": "set_alarm", "description": "Set an alarm.", "parameters": {"type": "object", "properties": {"hour": {"type": "integer"}, "minute": {"type": "integer"}, "volume": {"type": "number"}, "repeat": {"type": "boolean"}, "label": {"type": ["string", "null"]}}, "required": ["hour", "minute"]}}}], "raw_output": "<|channel|>analysis<|message|>Suche nach dem besten Café in Zürich.<|end|><|start|>assistant to=functions.search<|channel|>commentary json<|message|>{\"query\": \"café \\\"best\\\" Zürich ☕\", \"filters\": {\"tags\": [\"a\", \"b\"], \"max\": 3}}", "output_token_ids": [200005, 35644, 200008, 24726, 926, 5547, 2019, 29229, 58639, 306, 127483, 13, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 5701, 200008, 10848, 2975, 1243, 392, 66, 103112, 14927, 17844, 4017, 127483, 25701, 243, 672, 392, 50376, 1243, 10494, 27989, 1243, 9129, 64, 672, 392, 65, 17695, 392, 3228, 1243, 220, 18, 6478], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Suche nach dem besten Café in Zürich.", "tool_calls": [{"name": "search", "arguments": {"query": "café \"best\" Zürich ☕", "filters": {"tags": ["a", "b"], "max": 3}}}]}, "tags": ["single-call", "reasoning", "unicode", "nested-json", "string-escapes", "x-recipient-in-role"], "notes": "History render by the HF chat_template.jinja (recipient in the role header, content type written as plain ' json'). The template renders only tool_calls[0] of a message."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 5,
  "version": "0.5.20"
}

fail gpt-oss/llamacpp-commentary-content-only expected_match

Checks that did not pass for gpt-oss/llamacpp-commentary-content-only
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`one`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`special`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`token`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`rand:1:8`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`rand:2:8`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`rand:3:8`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`rand:4:8`	fail	content: expected "Hello, world!\nWhat's up?", got None
`expected_match`	`rand:5:8`	fail	content: expected "Hello, world!\nWhat's up?", got None

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-commentary-content-only --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 11 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,4 +1,4 @@
 {
-  "content": "Hello, world!\nWhat's up?",
+  "content": null,
   "reasoning_content": null,
   "tool_calls": []

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6404-L6405. Tags: no-call, x-preamble-commentary.

Raw output

<|channel|>commentary<|message|>Hello, world!
What's up?

Expected parse

{
  "content": "Hello, world!\nWhat's up?",
  "reasoning_content": null,
  "tool_calls": []
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-commentary-content-only", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6404-L6405", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>commentary<|message|>Hello, world!\nWhat's up?", "output_token_ids": [200005, 12606, 815, 200008, 13225, 11, 2375, 4175, 45350, 869, 30], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Hello, world!\nWhat's up?", "reasoning_content": null, "tool_calls": []}, "tags": ["no-call", "x-preamble-commentary"], "notes": "'Basic content only - commentary channel'."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200002,
    "kept_by_engine": false,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|return|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/llamacpp-recipient-in-channel-analysis expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/llamacpp-recipient-in-channel-analysis
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>'; tool_calls: expected ['special_function'], got []
`expected_match`	`one`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`token`	fail	reasoning_content: expected None, got '{"arg1": 1'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:1:8`	fail	reasoning_content: expected None, got '{"arg1":'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:2:8`	fail	reasoning_content: expected None, got '{"arg1": 1'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:3:8`	fail	reasoning_content: expected None, got '{"arg1": '; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:4:8`	fail	reasoning_content: expected None, got '{"arg1'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:5:8`	fail	reasoning_content: expected None, got '{"arg1":'; tool_calls: expected ['special_function'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1": 1'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1":'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1": 1'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1": '
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|channel\|>analysis to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None; reasoning_content: nonstream None, got '{"arg1":'
`split_invariance`	`*`	fail	vs one: token (fail): reasoning_content: one None, got '{"arg1": 1' \| rand:1:8 (fail): reasoning_content: one None, got '{"arg1":' \| rand:2:8 (fail): reasoning_content: one None, got '{"arg1": 1' \| rand:3:8 (fail): reasoning_content: one None, got '{"arg1": ' \| rand:4:8 (fail): reasoning_content: one None, got '{"arg1' \| rand:5:8 (fail): reasoning_content: one None, got '{"arg1":'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-recipient-in-channel-analysis --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 12 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|channel|>analysis to=functions.special_function<|message|>{\"arg1\": 1}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token, rand:2:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "reasoning_content": "{\"arg1\": 1",
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "reasoning_content": "{\"arg1\":",
+  "tool_calls": []
 }

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "reasoning_content": "{\"arg1\": ",
+  "tool_calls": []
 }

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "reasoning_content": "{\"arg1",
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6437-L6442. Tags: single-call, numeric-arguments, x-recipient-in-channel, x-call-on-analysis.

Raw output

<|channel|>analysis to=functions.special_function<|message|>{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-recipient-in-channel-analysis", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6437-L6442", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>analysis to=functions.special_function<|message|>{\"arg1\": 1}", "output_token_ids": [200005, 35644, 316, 28, 44580, 118760, 21094, 200008, 10848, 1170, 16, 1243, 220, 16, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-recipient-in-channel", "x-call-on-analysis"], "notes": "'Tool call with recipient in channel header'."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/llamacpp-recipient-in-channel-commentary expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/llamacpp-recipient-in-channel-commentary
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>'; tool_calls: expected ['special_function'], got []
`expected_match`	`one`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`token`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['special_function'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['special_function'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|channel\|>commentary to=functions.special_function<\|message\|>{"arg1": 1}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-recipient-in-channel-commentary --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 13 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|channel|>commentary to=functions.special_function<|message|>{\"arg1\": 1}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6451-L6456. Tags: single-call, numeric-arguments, x-recipient-in-channel, x-no-content-type.

Raw output

<|channel|>commentary to=functions.special_function<|message|>{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-recipient-in-channel-commentary", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6451-L6456", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": "<|channel|>commentary to=functions.special_function<|message|>{\"arg1\": 1}", "output_token_ids": [200005, 12606, 815, 316, 28, 44580, 118760, 21094, 200008, 10848, 1170, 16, 1243, 220, 16, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-recipient-in-channel", "x-no-content-type"], "notes": "'Tool call in commentary channel (channel header variant)'."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/llamacpp-recipient-in-role-constrain expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/llamacpp-recipient-in-role-constrain
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>'; tool_calls: expected ['special_function'], got []
`expected_match`	`one`	fail	content: expected None, got ' to=functions.special_function'; tool_calls: expected ['special_function'], got []
`expected_match`	`special`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; tool_calls: expected ['special_function'], got []
`expected_match`	`token`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; reasoning_content: expected None, got '{"arg1": 1'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:1:8`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; reasoning_content: expected None, got '{"'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:2:8`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; reasoning_content: expected None, got '{"arg1'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:3:8`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; reasoning_content: expected None, got '{"arg'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:4:8`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; tool_calls: expected ['special_function'], got []
`expected_match`	`rand:5:8`	fail	content: expected None, got ' to=functions.special_function to=functions.special_function'; reasoning_content: expected None, got '{"'; tool_calls: expected ['special_function'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function'
`stream_equals_nonstream`	`special`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'
`stream_equals_nonstream`	`token`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'; reasoning_content: nonstream None, got '{"arg1": 1'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'; reasoning_content: nonstream None, got '{"'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'; reasoning_content: nonstream None, got '{"arg1'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'; reasoning_content: nonstream None, got '{"arg'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream ' to=functions.special_function<\|channel\|>analysis <\|constrain\|>json<\|message\|>{"arg1": 1}<\|call\|>', got ' to=functions.special_function to=functions.special_function'; reasoning_content: nonstream None, got '{"'
`split_invariance`	`*`	fail	vs one: special (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function' \| token (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function'; reasoning_content: one None, got '{"arg1": 1' \| rand:1:8 (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function'; reasoning_content: one None, got '{"' \| rand:2:8 (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function'; reasoning_content: one None, got '{"arg1' \| rand:3:8 (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function'; reasoning_content: one None, got '{"arg' \| rand:4:8 (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function' \| rand:5:8 (fail): content: one ' to=functions.special_function', got ' to=functions.special_function to=functions.special_function'; reasoning_content: one None, got '{"'
`no_leakage`	`nonstream`	fail	content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/llamacpp-recipient-in-role-constrain --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 14 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.special_function<|channel|>analysis <|constrain|>json<|message|>{\"arg1\": 1}<|call|>",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.special_function",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: special, rand:4:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": " to=functions.special_function to=functions.special_function",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": " to=functions.special_function to=functions.special_function",
+  "reasoning_content": "{\"arg1\": 1",
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": " to=functions.special_function to=functions.special_function",
+  "reasoning_content": "{\"",
+  "tool_calls": []
 }

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": " to=functions.special_function to=functions.special_function",
+  "reasoning_content": "{\"arg1",
+  "tool_calls": []
 }

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
-  "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "arg1": 1
-      },
-      "name": "special_function"
-    }
-  ]
+  "content": " to=functions.special_function to=functions.special_function",
+  "reasoning_content": "{\"arg",
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6444-L6449. Tags: single-call, numeric-arguments, x-recipient-in-role, x-call-on-analysis.

Raw output

 to=functions.special_function<|channel|>analysis <|constrain|>json<|message|>{"arg1": 1}

Expected parse

{
  "content": null,
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "arg1": 1
      },
      "name": "special_function"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/llamacpp-recipient-in-role-constrain", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/ggml-org/llama.cpp/blob/7fe450e19305b828c199d602c23a8337aaa1f03b/tests/test-chat.cpp#L6444-L6449", "revision": "7fe450e19305b828c199d602c23a8337aaa1f03b", "license": "MIT", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright (c) 2023-2026 The ggml authors (MIT)."}, "tools": [{"type": "function", "function": {"name": "special_function", "parameters": {"type": "object", "properties": {"arg1": {"type": "integer", "description": "The arg."}}, "required": ["arg1"]}, "description": "I'm special"}}], "raw_output": " to=functions.special_function<|channel|>analysis <|constrain|>json<|message|>{\"arg1\": 1}", "output_token_ids": [316, 28, 44580, 118760, 21094, 200005, 35644, 220, 200003, 4108, 200008, 10848, 1170, 16, 1243, 220, 16, 92], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": null, "tool_calls": [{"name": "special_function", "arguments": {"arg1": 1}}]}, "tags": ["single-call", "numeric-arguments", "x-recipient-in-role", "x-call-on-analysis"], "notes": "'Tool call with constraint': the output starts right after the <|start|>assistant prompt."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/vllm-call-on-analysis-channel expected_match, stream_equals_nonstream, split_invariance, no_leakage

Checks that did not pass for gpt-oss/vllm-call-on-analysis-channel
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"location": "Paris'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"location":'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"location'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"location": "'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	reasoning_content: expected 'Thinking', got 'Thinking{"location":'; tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"location": "Paris'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"location":'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"location'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"location": "'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>analysis<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None; reasoning_content: nonstream 'Thinking', got 'Thinking{"location":'
`split_invariance`	`*`	fail	vs one: token (fail): reasoning_content: one 'Thinking', got 'Thinking{"location": "Paris' \| rand:1:8 (fail): reasoning_content: one 'Thinking', got 'Thinking{"location":' \| rand:2:8 (fail): reasoning_content: one 'Thinking', got 'Thinking{"location' \| rand:3:8 (fail): reasoning_content: one 'Thinking', got 'Thinking{"location": "' \| rand:4:8 (fail): reasoning_content: one 'Thinking', got 'Thinking{"' \| rand:5:8 (fail): reasoning_content: one 'Thinking', got 'Thinking{"location":'
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-call-on-analysis-channel --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 4 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>analysis<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|>",
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: token

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "reasoning_content": "Thinking{\"location\": \"Paris",
+  "tool_calls": []
 }

Strategies: rand:1:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "reasoning_content": "Thinking{\"location\":",
+  "tool_calls": []
 }

Strategies: rand:2:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "reasoning_content": "Thinking{\"location",
+  "tool_calls": []
 }

Strategies: rand:3:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "reasoning_content": "Thinking{\"location\": \"",
+  "tool_calls": []
 }

Strategies: rand:4:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
   "content": null,
-  "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "reasoning_content": "Thinking{\"",
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L580-L612. Tags: single-call, reasoning, x-recipient-in-role, x-call-on-analysis, x-constrain-nospace.

Raw output

<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>analysis<|constrain|>json<|message|>{"location": "Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": "Thinking",
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-call-on-analysis-channel", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L580-L612", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>analysis<|constrain|>json<|message|>{\"location\": \"Paris\"}", "output_token_ids": [200005, 35644, 200008, 133850, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 35644, 200003, 4108, 200008, 10848, 7693, 1243, 392, 72782, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Thinking", "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-call-on-analysis", "x-constrain-nospace"], "notes": "test_tool_call_split_across_deltas[analysis]: a functions.* recipient on the analysis channel is still a tool call. The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/vllm-call-then-final expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/vllm-call-then-final
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Done', got '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done'; tool_calls: expected ['search'], got []
`expected_match`	`one`	fail	tool_calls: expected ['search'], got []
`expected_match`	`special`	fail	tool_calls: expected ['search'], got []
`expected_match`	`token`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['search'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['search'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.search<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"query": "vllm"}<\|call\|>Done', got 'Done'
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-call-then-final --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 7 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Done",
+  "content": "<|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{\"query\": \"vllm\"}<|call|>Done",
   "reasoning_content": "Reasoning about query...",
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vllm"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": "Done",
   "reasoning_content": "Reasoning about query...",
-  "tool_calls": [
-    {
-      "arguments": {
-        "query": "vllm"
-      },
-      "name": "search"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718. Tags: single-call, reasoning, text-after-call, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{"query": "vllm"}<|call|><|start|>assistant<|channel|>final<|message|>Done

Expected parse

{
  "content": "Done",
  "reasoning_content": "Reasoning about query...",
  "tool_calls": [
    {
      "arguments": {
        "query": "vllm"
      },
      "name": "search"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-call-then-final", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L703-L718", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Reasoning about query...<|end|><|start|>assistant to=functions.search<|channel|>commentary<|constrain|>json<|message|>{\"query\": \"vllm\"}<|call|><|start|>assistant<|channel|>final<|message|>Done", "output_token_ids": [200005, 35644, 200008, 30377, 289, 1078, 5703, 1008, 200007, 200006, 173781, 316, 28, 44580, 16718, 200005, 12606, 815, 200003, 4108, 200008, 10848, 2975, 1243, 392, 85, 680, 76, 18583, 200012, 200006, 173781, 200005, 17196, 200008, 24537], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Done", "reasoning_content": "Reasoning about query...", "tool_calls": [{"name": "search", "arguments": {"query": "vllm"}}]}, "tags": ["single-call", "reasoning", "text-after-call", "x-recipient-in-role"], "notes": "test_cross_channel_with_tool. As in vllm-sequential-calls, text after <|call|> is only seen when the stop token is not applied."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200002,
    "kept_by_engine": false,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|return|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/vllm-preamble-call-end-terminated expected_match

Checks that did not pass for gpt-oss/vllm-preamble-call-end-terminated
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	content: expected 'Let me check the weather.', got 'Let me check the weather.{"location": "SF"}'; tool_calls: expected ['get_weather'], got []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-preamble-call-end-terminated --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 5 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream, one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": "Let me check the weather.",
+  "content": "Let me check the weather.{\"location\": \"SF\"}",
   "reasoning_content": null,
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "SF"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L490-L505. Tags: single-call, text-before-call, x-preamble-commentary, x-no-content-type, x-recipient-in-role.

Raw output

<|channel|>commentary<|message|>Let me check the weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|message|>{"location": "SF"}<|end|>

Expected parse

{
  "content": "Let me check the weather.",
  "reasoning_content": null,
  "tool_calls": [
    {
      "arguments": {
        "location": "SF"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-preamble-call-end-terminated", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L490-L505", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>commentary<|message|>Let me check the weather.<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|message|>{\"location\": \"SF\"}<|end|>", "output_token_ids": [200005, 12606, 815, 200008, 12845, 668, 2371, 290, 11122, 13, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200008, 10848, 7693, 1243, 392, 46661, 18583, 200007], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": "Let me check the weather.", "reasoning_content": null, "tool_calls": [{"name": "get_weather", "arguments": {"location": "SF"}}]}, "tags": ["single-call", "text-before-call", "x-preamble-commentary", "x-no-content-type", "x-recipient-in-role"], "notes": "test_commentary_with_recipient_excluded. The call message is closed by <|end|> instead of the <|call|> stop token."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

fail gpt-oss/vllm-sequential-calls expected_match, stream_equals_nonstream, no_leakage, parallel_order

Checks that did not pass for gpt-oss/vllm-sequential-calls
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get...; tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather', 'get_time'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|><\|start\|>assistant to=functions.get..., got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'
`parallel_order`	`nonstream`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`one`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`special`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`token`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`rand:1:8`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`rand:2:8`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`rand:3:8`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`rand:4:8`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []
`parallel_order`	`rand:5:8`	fail	expected 2 calls ['get_weather', 'get_time'], got 0 []

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-sequential-calls --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 6 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,18 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{\"timezone\": \"UTC\"}<|call|>",
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "timezone": "UTC"
-      },
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,17 +2,4 @@
   "content": null,
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    },
-    {
-      "arguments": {
-        "timezone": "UTC"
-      },
-      "name": "get_time"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Provenance: engine_test, https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745. Tags: parallel-calls, reasoning, x-sequential-calls, x-recipient-in-role.

Raw output

<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{"location": "Paris"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{"timezone": "UTC"}

Expected parse

{
  "content": null,
  "reasoning_content": "Thinking",
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    },
    {
      "arguments": {
        "timezone": "UTC"
      },
      "name": "get_time"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-sequential-calls", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L723-L745", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}, {"type": "function", "function": {"name": "get_time", "parameters": {"type": "object", "properties": {"timezone": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|><|start|>assistant to=functions.get_time<|channel|>commentary<|constrain|>json<|message|>{\"timezone\": \"UTC\"}", "output_token_ids": [200005, 35644, 200008, 133850, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200003, 4108, 200008, 10848, 7693, 1243, 392, 72782, 18583, 200012, 200006, 173781, 316, 28, 44580, 775, 6425, 200005, 12606, 815, 200003, 4108, 200008, 10848, 106775, 1243, 392, 32674, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Thinking", "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}, {"name": "get_time", "arguments": {"timezone": "UTC"}}]}, "tags": ["parallel-calls", "reasoning", "x-sequential-calls", "x-recipient-in-role"], "notes": "test_tool_index_across_calls: the two deltas concatenated. Harmony has no parallel-call wrapper; consecutive call messages are separated by <|call|>, which is a stop token, so this output only reaches a parser when the stop is not applied (e.g. ignore_eos, vLLM #50690). The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 2,
  "version": "0.5.20"
}

fail gpt-oss/vllm-split-deltas-constrain-nospace expected_match, stream_equals_nonstream, no_leakage

Checks that did not pass for gpt-oss/vllm-split-deltas-constrain-nospace
Check	Strategy	Result	Detail
`expected_match`	`nonstream`	fail	content: expected None, got '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>'; tool_calls: expected ['get_weather'], got []
`expected_match`	`one`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`special`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`token`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:1:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:2:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:3:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:4:8`	fail	tool_calls: expected ['get_weather'], got []
`expected_match`	`rand:5:8`	fail	tool_calls: expected ['get_weather'], got []
`stream_equals_nonstream`	`one`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`special`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`token`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:1:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:2:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:3:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:4:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`stream_equals_nonstream`	`rand:5:8`	fail	content: nonstream '<\|start\|>assistant to=functions.get_weather<\|channel\|>commentary<\|constrain\|>json<\|message\|>{"location": "Paris"}<\|call\|>', got None
`no_leakage`	`nonstream`	fail	content contains '<\|start\|>'; content contains '<\|message\|>'; content contains '<\|channel\|>'; content contains '<\|constrain\|>'; content contains '<\|call\|>'

Minimal repro

uv run canitoolcall run --engine sglang --fixtures fixtures/gpt-oss/imported.jsonl --id gpt-oss/vllm-split-deltas-constrain-nospace --strategy one --strategy rand:1:8 --strategy rand:2:8 --strategy rand:3:8 --strategy rand:4:8 --strategy rand:5:8 --strategy special --strategy token --observed all

Set up the engine first with scripts/engines/sglang.sh; this run used sglang 0.5.20. The fixture is line 3 of fixtures/gpt-oss/imported.jsonl.

Observed vs expected

Identical parses are grouped. Empty strings are shown as null, as in strict comparison.

Strategies: nonstream

--- expected
+++ observed
@@ -1,12 +1,5 @@
 {
-  "content": null,
+  "content": "<|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}<|call|>",
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Strategies: one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

--- expected
+++ observed
@@ -2,11 +2,4 @@
   "content": null,
   "reasoning_content": "Thinking",
-  "tool_calls": [
-    {
-      "arguments": {
-        "location": "Paris"
-      },
-      "name": "get_weather"
-    }
-  ]
+  "tool_calls": []
 }

Fixture

Raw output

<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{"location": "Paris"}

Expected parse

{
  "content": null,
  "reasoning_content": "Thinking",
  "tool_calls": [
    {
      "arguments": {
        "location": "Paris"
      },
      "name": "get_weather"
    }
  ]
}

Fixture record (JSONL, ready to vendor into an engine's tests)

{"id": "gpt-oss/vllm-split-deltas-constrain-nospace", "family": "gpt-oss", "models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"], "spec_version": "0.1", "provenance": {"kind": "engine_test", "source_url": "https://github.com/vllm-project/vllm/blob/ced6857afa0ea7b2e3f0846a62e1394e90f15607/tests/parser/test_harmony.py#L580-L612", "revision": "ced6857afa0ea7b2e3f0846a62e1394e90f15607", "license": "Apache-2.0", "generator": "scripts/fixtures/gpt-oss/import_gpt_oss.py", "attribution": "Copyright contributors to the vLLM project (Apache-2.0)."}, "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], "raw_output": "<|channel|>analysis<|message|>Thinking<|end|><|start|>assistant to=functions.get_weather<|channel|>commentary<|constrain|>json<|message|>{\"location\": \"Paris\"}", "output_token_ids": [200005, 35644, 200008, 133850, 200007, 200006, 173781, 316, 28, 44580, 775, 170154, 200005, 12606, 815, 200003, 4108, 200008, 10848, 7693, 1243, 392, 72782, 18583], "tokenizer": {"repo": "openai/gpt-oss-20b", "revision": "6cee5e81ee83917806bbde320786a8fb61efebee", "mode": "hf"}, "expected": {"content": null, "reasoning_content": "Thinking", "tool_calls": [{"name": "get_weather", "arguments": {"location": "Paris"}}]}, "tags": ["single-call", "reasoning", "x-recipient-in-role", "x-constrain-nospace"], "notes": "test_tool_call_split_across_deltas[commentary]: the two deltas of the test, concatenated. The trailing stop token <|call|> was removed (raw_output ends before the stop token)."}

Parser configuration

{
  "auto_detected": {
    "reasoning_parser": "gpt-oss",
    "tool_call_parser": "gpt-oss"
  },
  "chat_encoding_spec": null,
  "chat_template_kwargs": null,
  "chat_template_sha256": "a4c9919cbbd4acdd51ccffe22da049264b1b73e59055fa58811a99efbd7c8146",
  "detokenizer": "DetokenizerManager._decode_batch_token_id_output",
  "engine": "sglang",
  "hf_config": {
    "architectures": [
      "GptOssForCausalLM"
    ],
    "model_type": "gpt_oss"
  },
  "hf_config_error": null,
  "model": "openai/gpt-oss-20b",
  "no_stop_trim": false,
  "notes": [],
  "prompt_tail": {
    "ids": [
      200006,
      173781
    ],
    "source": "generation_prompt"
  },
  "reasoning_detector": "GptOssDetector",
  "reasoning_effort": null,
  "reasoning_enabled": true,
  "reasoning_parser": "gpt-oss",
  "separate_reasoning": true,
  "skip_special_tokens": false,
  "spaces_between_special_tokens": true,
  "stop": {
    "appended": true,
    "finish_reason": "stop",
    "id": 200012,
    "kept_by_engine": true,
    "rule": "harmony: <|call|> after a message with a recipient, else <|return|>",
    "token": "<|call|>"
  },
  "stream_reasoning": true,
  "template_force_reasoning": true,
  "template_reasoning_config": "ReasoningToggleConfig(toggle_param=None, default_enabled=None, special_case='always', effort_kwarg=None)",
  "thinking": null,
  "tokenizer": {
    "class": "TokenizersBackend",
    "loader": "sglang.srt.utils.hf_transformers_utils.get_tokenizer(revision=, tokenizer_revision=)",
    "repo": "openai/gpt-oss-20b",
    "revision": "6cee5e81ee83917806bbde320786a8fb61efebee",
    "trust_remote_code": false
  },
  "tokenizer_mode": "hf",
  "tool_call_detector": "GptOssDetector",
  "tool_call_parser": "gpt-oss",
  "tool_choice": "auto",
  "tools_offered": 1,
  "version": "0.5.20"
}

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model