Skip to content

fix: Claude 5.x compaction lifecycle (replay gate, per-iteration cost) with end-to-end tests - #4688

Merged
aheritier merged 6 commits into
mainfrom
test/claude5x-compaction-lifecycle
Oct 21, 2026
Merged

aheritier merged 6 commits into
mainfrom
test/claude5x-compaction-lifecycle

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

Part of #4654 (PR 2 of 3). Part of #4583 and epic #4580.

What

Two compaction fixes, then two end-to-end tests that pin the whole compaction lifecycle on Claude 5.x: compaction, a SQLite reload, and the next request.

Fixes

Commit Bug Fix
fix(anthropic): replay native compaction only where it is enabled The replay check compared only the provider-state name, which the Claude API and Vertex clients share (anthropic). After a native compaction on the Claude API, a client without native_compaction (Vertex, or a Claude API model without the option) still sent the signed block and the compact-2026-09-04 beta. A client replays the block only when its own provider_opts.native_compaction is set. Otherwise it sends the summary text that every summary message already carries.
fix(runtime): price native compaction iterations separately The compaction cost was computed on the summed usage, so a long-context tier was chosen on the total, not per request. chat.CompactionResult.Iterations keeps each reported iteration. Each one is priced on its own and the costs are summed. If any iteration can't be priced, the cost is nil, as for any other unpriced call.

Tests

Test Pins Mutation that makes it fail
TestCompaction_NotReplayedWithoutNativeCompaction, TestVertexDoesNotReplayCompactionBlock No block and no beta without the option, on the Claude API and on Vertex The fix itself (written first, red, then green)
TestNativeCompactionPricesEachIteration, TestCompactionCostUnpricedIteration Tiered rates chosen per iteration; nil on an unpriced iteration The fix itself
TestClaude5LocalCompactionLifecycle (pkg/runtime) Threshold compaction summarizes the long history and keeps the signed thinking turn and the latest question. After a SQLite reload, the next request carries the summary, that turn's signature, and the drop_block binding Loading FirstKeptEntry as 0; dropping both ProviderState and ThinkingSignature on load
TestClaude5NativeCompactionLifecycle (pkg/runtime) Native compaction stores the signed block, both iterations and their cost. After a reload, the next request starts with the block, verbatim, and sends the beta Forcing the native_compaction gate off

Both end-to-end tests use a real Sonnet 5.5 client built by the provider registry, a small scripted SSE server (anthropictest, ~150 lines), and the SQLite store. Each takes ~0.3s under -race.

Not added, and why

  • Per-iteration tier choice end to end: the runtime test uses flat test rates, so it pins that iteration costs reach the stored summary. Choosing a tier per iteration is pinned by the unit test, which is cheaper.
  • Bedrock compaction lifecycle: it would need a Bedrock event-stream fake server, which test(runtime): Claude 5.x lifecycle — compaction, reload, switches, shared clients #4654 rules out.
  • Switching models and reloading at request level: PR 3.

Notes

  • Dropping ProviderState alone doesn't break the local lifecycle test: the Anthropic converter rebuilds the signed thinking block from ReasoningContent + ThinkingSignature. Both have to be lost for the signature to be lost.
  • The on-demand compaction page now lists Google Cloud (beta) among its platforms. docker-agent still rejects native_compaction on Vertex; lifting that is out of scope here.

References

Change Documentation
A request that carries a compaction block needs the compact-2026-09-04 beta header ("Send the compact-2026-09-04 beta header … on every later request that carries the signed block"); without it, a 400 says compaction is not one of the expected content block types Compaction on demand
Keep the block exactly as returned, including its signature, and send it first on the next request Compaction on demand
"The summarization call is billed … and usage.iterations reports it … sum across usage.iterations, not the top-level fields" Compaction on demand
Haiku 5.5 "is priced by prompt length … Each request is priced on its own", so each compaction iteration picks its own tier Claude pricing
A kept thinking block is sent back with its signature; thinking.block_binding.prefix_mismatch_behavior: "drop_block" drops it, rather than failing, when the prefix changed Preserved thinking

Validation

  • task build passes.
  • task test passes.
  • golangci-lint v2.14.0 reports 0 issues, and go run ./lint . reports no offenses.
  • go test -race -shuffle=on ./pkg/runtime/ ./pkg/model/provider/anthropic/... ./pkg/chat/ passes.
  • An independent review found no critical or important issues. Three minor findings are fixed here:
    • a test comment overstated what the flat-rate cost assertion pins;
    • a raw-JSON assertion now counts messages instead;
    • the provider docs no longer say every Anthropic client replays the block.

Deferred minor findings:

  • The native_compaction option name is written as a literal in the Anthropic and Vertex packages and as a constant in the runtime.
  • The internal Anthropic test helpers (newScriptedServer, textReply, compactionReply) could move onto anthropictest.

The replay check only compared the provider name, which the Claude API
and Vertex clients share. After a native compaction on the Claude API,
a client without native_compaction (Vertex, which rejects the option,
or a Claude API model without it) sent the signed block and the
compaction beta. Only a client configured with native_compaction now
replays the block; others send the summary text every summary message
carries.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Native compaction summed every iteration's usage and priced the total,
so a context tier was chosen on the sum: two 60k iterations of Haiku 5.5
were priced as one 120k request at the higher rate. Keep each iteration
and price it on its own; an unpriced iteration leaves the compaction
unpriced.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
A thinking turn kept by threshold compaction is replayed with its
signature, after the summary and under the drop_block binding, once the
session is reloaded from SQLite.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
The signed compaction block and its per-iteration usage are stored with
the summary; after a reload the next request replays the block verbatim
with the compaction beta.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
@aheritier
aheritier requested a review from a team as a code owner October 21, 2026 11:55
Copilot AI balanced review requested due to automatic review settings October 21, 2026 11:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking correctness issues remain; validation results were reported but not independently rerun.

1 open issue
What changed in this PR

Fixes native compaction replay and billing for Claude clients, with lifecycle coverage for compaction, SQLite reload, and continuation.

Changes:

  • Gate signed-block replay on native_compaction.
  • Preserve and price compaction usage per iteration.
  • Add provider regression tests and runtime lifecycle tests.
File Description
pkg/​runtime/​native_compaction.go Prices iterations separately.
pkg/​runtime/​native_compaction_test.go Tests tiered and unpriced costs.
pkg/​runtime/​claude5_compaction_lifecycle_test.go Tests compaction, reload, and continuation.
pkg/​model/​provider/​anthropic/​vertex/​vertex_test.go Tests summary fallback on Vertex.
pkg/​model/​provider/​anthropic/​features_integration_test.go Enables compaction in replay coverage.
pkg/​model/​provider/​anthropic/​compaction.go Gates replay and captures iterations.
pkg/​model/​provider/​anthropic/​compaction_test.go Tests replay gating and iteration usage.
pkg/​model/​provider/​anthropic/​client.go Uses client-specific replay eligibility.
pkg/​model/​provider/​anthropic/​beta_converter.go Applies the replay gate during conversion.
pkg/​model/​provider/​anthropic/​beta_client.go Gates the compaction beta header.
pkg/​model/​provider/​anthropic/​anthropictest/​server.go Adds scripted responses and request capture.
pkg/​model/​provider/​anthropic/​anthropictest/​server_test.go Tests the scripted server.
pkg/​chat/​compaction.go Stores and clones iteration usage.
docs/​providers/​anthropic/​index.md Clarifies replay eligibility.

Review effort: Balanced

Comment thread pkg/model/provider/anthropic/compaction.go
@aheritier aheritier added area/docs Documentation changes area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Oct 21, 2026
With replay gated on native_compaction, the edited-summary and
foreign-provider tests passed on a client without the option whatever
ReplayableCompaction did. Enable the option so they check it again.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
@aheritier
aheritier enabled auto-merge October 21, 2026 12:11
@aheritier
aheritier added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit f17bcb0 Oct 21, 2026
22 checks passed
@aheritier
aheritier deleted the test/claude5x-compaction-lifecycle branch October 21, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants