Skip to content

fix(claude): explicit Claude 5.x model table for request policy - #4657

Merged
aheritier merged 18 commits into
mainfrom
feat/claude5x-model-table
Oct 20, 2026
Merged

aheritier merged 18 commits into
mainfrom
feat/claude5x-model-table

Conversation

@aheritier

@aheritier aheritier commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Supersedes #4655, same commits, moved to a docker/docker-agent branch so #4656 can stack on it. The earlier review threads (Copilot) and their answers stay on #4655.

Closes #4651. Part of #4583 (Claude 5.x core) and epic #4580.

What

Replace the name-based guesses for Claude 5.x with an explicit table of the nine models (modelinfo.LookupClaude5). Anthropic Direct, Vertex and Bedrock Converse now send each of them a request it accepts. IDs outside the table keep exactly today's behavior.

Model Can disable thinking Server prefix check
Haiku 5.5, Opus 5, Sonnet 5 yes Haiku 5.5 only
Opus 5.5, Sonnet 5.5, Fable 5.1 no yes
Fable 5, Mythos 5, Mythos 5.1 no no

All nine use adaptive thinking by default, reject token budgets and sampling params, and accept every effort level up to max.

Bugs fixed (each with a test that failed before)

  • Haiku 5.5: thinking_budget: high was sent as budget_tokens, which every 5.x model rejects.
  • Bedrock: effort and token budgets became budget_tokens on Sonnet 5/5.5, Haiku 5.5, Fable and Mythos.
  • Bedrock: temperature/top_p/top_k were sent to 5.x models.
  • thinking_budget: none never reached the Claude clients through the factory. applyModelDefaults turns it into a nil budget plus WithNoThinking, so on 5.x thinking stayed on at the server default, on Direct, Vertex and Bedrock. It is now sent as disabled where the model allows it and as adaptive/low otherwise.
  • Thinking prefix check: it was wrong for Mythos 5.1 (no server check) and missing for Haiku, Opus and Sonnet 5.5.
  • Opus 5.5 and Sonnet 5.5 were sent thinking: disabled, which they reject.
  • TUI effort cycle hid xhigh/max on Haiku 5.5, Sonnet 5 and Sonnet 5.5.

Behavior changes to review

  • none on a model that can't disable thinking (Opus/Sonnet 5.5, Fable, Mythos) is sent as adaptive with low effort. When the value came from config, a startup warning says so. Internal calls (titles, MCP sampling) get the same mapping silently.
  • A token budget on a 5.x model becomes adaptive with no effort, so the model's own default applies. max_tokens is not raised, and the warning now says the token ceiling is not enforced.
  • A request ending on an assistant message (prefill) is rejected locally before sending, on Direct, Vertex and Bedrock.
  • Tool-mode structured output on 5.x fails with ErrorCodeStructuredOutputFailed after the first plain-text answer, instead of retrying with reminders. Each retry would end on the model's answer, which 5.x rejects. feat(Claude): Claude 5.x core — request policy and signed history #4583 asks for "one inference, then a graceful model error", with no role rewrite.
  • Mythos 5.1 no longer gets the automatic drop_block binding. Haiku, Opus and Sonnet 5.5 now get it, the same way Fable 5.1 already did. The default policy itself (drop_block vs error) is feat(anthropic): Claude 5.x safety defaults (prefix error, temporary fallback block, Mythos 5 on Converse) #4652.

Tests

  • Table lookup: fixtures written independently of the production table, covering Vertex @date, Bedrock us.anthropic.…-v1:0, gateway prefixes and near-miss IDs. openai/claude-… IDs deliberately don't match.
  • Final requests captured through the real factory (providers.NewDefaultRegistry()) for Direct standard, Direct Beta, Vertex Model Garden (path, anthropic_version, bearer auth, no API key, de-duplicated betas), Bedrock Converse and a title clone, plus legacy controls.
  • Runtime test for the structured-output path.

References

Vendor documentation behind each change (all links checked on 2026-10-20). The per-model matrix these rules come from is in #4583.

Change Documentation
The nine models and their IDs; dateless 5.x IDs are pinned snapshots Model IDs and versions, Migration guide
Adaptive thinking by default; manual budget_tokens rejected (token budgets coerced to adaptive) Extended thinking
Effort levels low … max, including xhigh (TUI effort cycle) Effort
Which models can disable thinking; sampling parameters rejected; final assistant prefill rejected Per-model migration guides: Haiku 5.5, Opus 5.5, Sonnet 5.5, Fable 5.1 / Mythos 5.1
Server prefix check per model; drop_block binding Preserved thinking
Vertex request shape (model in URL, vertex-2023-10-16, OAuth) Claude on Vertex AI
Bedrock Converse: adaptive thinking fields; per-model limits AWS: adaptive thinking, AWS model cards

Validation

  • task build passes.
  • golangci-lint v2.14.0 reports 0 issues, and go run ./lint . reports no offenses.
  • go test -race -shuffle=on ./pkg/modelinfo/ ./pkg/model/provider/... ./pkg/runtime/ passes.
  • go test ./e2e -run '^TestWasmProviderDependencies$|^TestDependencies$' passes.
  • task test-wasm-providers: the WASM JS packages pass. Running it needed a minimal environment, because Node rejected my shell's environment size.
  • task test fails only pkg/teamloader TestLoadExamples (10 Gemini examples, GOOGLE_API_KEY or GEMINI_API_KEY environment variable is required under .env.test). The base commit 7eedf8d fails identically on this machine.

Not verified against live services: Bedrock thinking: {"type":"disabled"} and the thinking-binding beta on Vertex. The docs say so.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…bilities

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
… table

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…actory

applyModelDefaults turns a configured none into a nil budget plus
WithNoThinking, so the Anthropic and Bedrock clients never saw it and
Claude 5.x kept its default thinking. Restore the explicit none for the
Claude 5.x table so it is sent as disabled or adaptive low.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…orts

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…retry

A tool-mode reminder retry ends the request on the model's plain-text
answer, which Claude 5.x rejects as an assistant prefill. Fail with the
coded structured-output error after the first inference instead.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…and 0 budgets

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
…e thinking

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Importing vertexai for IsModelGardenConfig linked the OpenAI SDK into the
lean embedder surface (e2e TestDependencies). Check the publisher directly.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
… the factory level

Signed-off-by: Arnaud Héritier <aheritier@apache.org>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-transport changes need human validation of the explicitly unverified Bedrock disabled-thinking and Vertex thinking-binding behavior.

No open issues 🎉

What changed in this PR

Adds an explicit nine-model Claude 5.x policy table for Anthropic, Vertex and Bedrock, while preserving existing behavior for other model IDs.

Changes:

  • Aligns thinking, effort, sampling and prefix-check behavior with model capabilities.
  • Rejects assistant prefills locally and avoids unsupported structured-output retries.
  • Adds request-policy tests and updates provider documentation.
File Description
pkg/​runtime/​structured_output.go Stops unsupported Claude 5.x reminder retries.
pkg/​runtime/​structured_output_test.go Tests failure after one inference.
pkg/​runtime/​loop.go Passes serving model to structured-output handling.
pkg/​modelinfo/​thinking_levels.go Exposes all supported effort levels.
pkg/​modelinfo/​modelinfo.go Uses table-based thinking capabilities.
pkg/​modelinfo/​claude5.go Defines model policies and ID lookup.
pkg/​modelinfo/​claude5_test.go Tests lookup and capabilities.
pkg/​model/​provider/​providerutil/​prefill.go Detects final assistant prefills.
pkg/​model/​provider/​providerutil/​prefill_test.go Tests prefill detection.
pkg/​model/​provider/​providers/​claude5_requests_test.go Captures factory-built requests across transports.
pkg/​model/​provider/​factory.go Warns when configured thinking cannot be disabled.
pkg/​model/​provider/​factory_test.go Tests warning conditions.
pkg/​model/​provider/​bedrock/​client.go Applies Claude 5.x Converse request rules.
pkg/​model/​provider/​bedrock/​client_test.go Tests Bedrock policy handling.
pkg/​model/​provider/​anthropic/​thinking.go Corrects disabled-thinking and effort handling.
pkg/​model/​provider/​anthropic/​thinking_binding_test.go Updates binding expectations and warning tests.
pkg/​model/​provider/​anthropic/​system_context_test.go Uses a legacy model for prefill coverage.
pkg/​model/​provider/​anthropic/​features_integration_test.go Tests Claude 5.x requests and legacy behavior.
pkg/​model/​provider/​anthropic/​client.go Rejects Claude 5.x prefills locally.
pkg/​model/​provider/​anthropic/​capabilities.go Prioritizes explicit model policies.
pkg/​model/​provider/​anthropic/​capabilities_test.go Tests all nine model policies.
docs/​providers/​bedrock/​index.md Documents Converse behavior and verification limits.
docs/​providers/​anthropic/​index.md Documents Claude 5.x request rules.
agent-schema.json Updates prefix-mismatch policy documentation.

Review effort: Balanced

@aheritier
aheritier added this pull request to stack #4659 October 19, 2026 22:25
@aheritier aheritier added area/docs Documentation changes area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/providers/bedrock AWS Bedrock provider support area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection labels Oct 19, 2026
@aheritier aheritier added the kind/fix PR fixes a bug (maps to fix:). Use on PRs only. label Oct 19, 2026
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
@aheritier
aheritier added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit af89f74 Oct 20, 2026
22 checks passed
@aheritier
aheritier deleted the feat/claude5x-model-table branch October 20, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/providers/bedrock AWS Bedrock provider support area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(anthropic): explicit Claude 5.x model table and request-policy resolver

3 participants