Qwen3 (Hermes-style JSON tool calls) on transformers 5.17.0

unsupported 50 unsupported

Run

Engine

transformers 5.17.0

Run

2026-09-26T05:18:44Z to 2026-09-26T05:18:47Z

Platform

linux-x86_64, Python 3.12.14, canitoolcall 0.1.0.dev0

Strategies

one, special, token, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8 (normalization soft-v1); not counted for this engine: one, special, rand:1:8, rand:2:8, rand:3:8, rand:4:8, rand:5:8

Fixtures digest

a1e7b55367e195e8aa6e2ef746b2cc141b44f37caad0887ff8f30b104616f836 the fixture corpus used for this site matches the run

Engine details

{
  "huggingface_hub": "1.33.0",
  "jinja2": "3.1.6",
  "jmespath": "1.1.0",
  "pinned_repos": {
    "google/gemma-4-12B-it": "707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7",
    "google/gemma-4-26B-A4B-it": "4d7ae4984b7db7de8f8457170b3f1a419ee76d52",
    "google/gemma-4-31B-it": "842da3794eaa0b77d5f08bae87a17459d91ff475",
    "google/gemma-4-E2B-it": "3e22461f65e89153144f8adb70e3b8c2cc9845a7",
    "google/gemma-4-E4B-it": "ee0ef6023621cff504d758262d4e04895a5af4a2"
  },
  "pinned_version": "5.17.0",
  "tokenizers": "0.23.2",
  "tokens_per_step": "one",
  "torch": null
}

Results file

data/transformers-5.17.0.json

Checks

No checks applied (no supported fixtures).

Fixtures needing attention

Every supported fixture passed strictly on every realistic strategy.

Unsupported fixtures

The adapter declined these fixtures because this engine version has no parser for the family or model.

qwen3-hermes/bug-sglang-30480-truncated-at-opener: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/bug-sglang-30480-truncated-mid-arguments: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/empty-arguments-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/empty-arguments-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/instruct2507-empty-arguments: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/instruct2507-no-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/instruct2507-parallel-calls: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/instruct2507-single-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/instruct2507-text-before-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/json-string-argument: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/long-arguments: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/marker-in-arguments: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/multi-turn-final-answer-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/multi-turn-second-call-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/nested-json-deep: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/no-call-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/no-call-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/numeric-arguments: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/parallel-mixed-tools-text-before-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/parallel-three-calls-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/parallel-two-calls-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/sglang-malformed-json-in-tags: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/sglang-qwen25-four-parallel-calls: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/sglang-text-before-call-with-space: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/single-call-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/single-call-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/string-escapes-code: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/text-before-call-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/text-before-call-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-multi-turn: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-no-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-parallel-calls: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-single-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-text-before-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/thinking2507-truncated-in-reasoning: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/truncated-after-open-tag: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/truncated-in-reasoning: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/truncated-mid-arguments-no-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/truncated-mid-arguments-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/truncated-second-parallel-call: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/unicode-emoji-thinking: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-bug-19056-boolean-argument: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-content-and-call-single-chunk: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-invalid-json-missing-brace: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-no-newlines-no-spaces: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-plain-text-forwarded: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-reasoning-multiline: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-reasoning-without-start-token: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-text-then-call-no-separator: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)
qwen3-hermes/vllm-two-calls-no-separator: no Hub repo of family 'qwen3-hermes' ships a response_template in tokenizer_config.json (transformers only parses with a repo-shipped template)

pass strict match on every realistic strategy
soft pass only whitespace differs (normalization soft-v1)
fail a check failed
error the harness failed, not the engine's parser
unsupported the engine has no parser for this family or model