Skip to content

test: pin Claude 5.x requests after a model or transport switch and reload - #4689

Open
aheritier wants to merge 2 commits into
mainfrom
test/claude5x-switch-requests
Open

aheritier wants to merge 2 commits into
mainfrom
test/claude5x-switch-requests

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

Closes #4654 (PR 3 of 3, after #4686 and #4688). Part of #4583 and epic #4580.

What

Test-only. This PR pins what a reloaded Claude 5.x conversation sends after a model or transport switch, and adds the one compaction safeguard case that was missing.

Test Pins Mutation that makes it fail
TestClaude5SwitchAfterReload (pkg/model/provider/providers) A tool turn recorded by Sonnet 5.5 on the Claude API is saved and reloaded (JSON round trip). The next request keeps its signed thinking: verbatim, under drop_block and with thinking-binding-controls, on the same model, on Haiku 5.5 and on Vertex; as signed reasoningContent on Bedrock. The raw blocks are replayed, not rebuilt: a redacted_thinking block that exists only in the raw state reaches the Claude API and Vertex. The new PDF attachment rides on the new user turn on every transport Disabling both raw replay paths (ReplayableProviderState and own-raw recovery); stripping thinking when the recorded model differs from the request model; dropping the signature in Bedrock's fallback path
TestCompactConversation_Safeguards/refusal (pkg/model/provider/anthropic) Documents the refusal the API describes (200, empty content, stop_reason: refusal): it takes the same unexpected-stop-reason error as an end_turn answer, so the runtime keeps the context (TestNativeCompactionErrorLeavesTranscriptIntact) — (documents current behavior)

The switch test reuses the request-capture harness from claude5_requests_test.go, now able to send a whole conversation. There is no runtime and no fake server, and each case takes milliseconds.

Why "keep sending" is the correct behavior

The docs say to "keep sending the full history on every request, thinking blocks included, and let the API drop what the current model can't read". The reasoning is lost for good only if the client strips it. Sonnet 5.5 and Haiku 5.5 blocks are also bound to the account that produced them; another account's request succeeds and the blocks are dropped. Replay in docker-agent never checks which model recorded a turn, and this test keeps it that way.

Results table

The full model × transport × feature table is posted on #4654. In short:

  • the Claude API is covered end to end, including both compaction lifecycles;
  • Vertex and Bedrock are covered at request and converter level;
  • 🚫 expected refusals: native compaction on Vertex and Bedrock, Mythos 5 outside Bedrock Mantle, and edited thinking inside the active tool loop;
  • ❔ not established:

Not added, and why

Notes

  • Without a models.dev store, Bedrock treats a model as text-only and drops the PDF. Production always passes the store (team loader, model switcher), so the Bedrock case uses the embedded snapshot through a stubbed fetcher, with no network.
  • Follow-up to consider: Google Cloud now documents on-demand compaction (beta), while docker-agent rejects native_compaction on Vertex.

References

Behavior Documentation
Keep sending thinking across model switches; the API drops what the current model can't read Preserved thinking, switching models mid-conversation
Sonnet 5.5 / Haiku 5.5 thinking is bound to the account that produced it; thinking-binding-controls-2026-08-01 reports drops on the Claude API and Google Cloud Preserved thinking, account-bound thinking
The prefix check covers system, tools and earlier messages; drop_block drops rather than rejects Preserved thinking
A refused compaction returns a 200 with empty content; check stop_reason Compaction on demand

Validation

  • task build passes.

  • task test passes.

  • golangci-lint v2.14.0 reports 0 issues, and go run ./lint . reports no offenses.

  • go test -race -shuffle=on ./pkg/model/provider/... passes.

  • An independent review found no critical issues and three important ones, all fixed here:

    • the "verbatim" claim wasn't pinned, since the flattened fallback rebuilt the same blocks; a raw-only redacted_thinking block now pins it;
    • a compact-* beta assertion could never fail, and is removed;
    • the catalogue store now uses modelsdev.NewDatabaseStore, as elsewhere.

    Minor findings fixed: a realistic Version/MessageID on the fixture, checked type assertions, the attachment checked on the new turn, and the refusal commit's wording.

A reloaded turn recorded by Sonnet 5.5 on the Claude API keeps its
signed thinking on the same model, on Haiku 5.5 and on Vertex (verbatim,
under the drop_block binding), and on Bedrock as signed reasoning; the
new attachment reaches every transport. The API drops thinking the new
model or account cannot read; stripping it client-side would lose it.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
The API answers a refused compaction with a 200, empty content and
stop_reason refusal. It takes the same unexpected-stop-reason error as
an end_turn answer, so the runtime keeps the context.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
@aheritier
aheritier requested a review from a team as a code owner October 21, 2026 16:01
Copilot AI balanced review requested due to automatic review settings October 21, 2026 16:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review found no blocking issues in these focused test-only changes; tests were not executed during review.

No open issues 🎉

What changed in this PR

Adds test-only coverage for Claude 5.x history replay after reload and model or transport switches, completing the switch-testing portion of #4654.

Changes:

  • Checks signed thinking and attachment requests across Claude API, Vertex, and Bedrock.
  • Extends request capture to accept full conversations.
  • Adds a native-compaction refusal safeguard case.
File Description
pkg/​model/​provider/​providers/​claude5_switch_requests_test.go Tests reloaded history across model and transport switches.
pkg/​model/​provider/​providers/​claude5_requests_test.go Supports full-history request capture.
pkg/​model/​provider/​anthropic/​compaction_test.go Checks compaction refusal error handling.

Review effort: Balanced

@aheritier
aheritier enabled auto-merge October 21, 2026 16:06
@aheritier aheritier added area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models kind/test Test-only changes labels Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models kind/test Test-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(runtime): Claude 5.x lifecycle — compaction, reload, switches, shared clients

2 participants