Skip to content

test(claude): pin Haiku 5.5 price tier, reload pricing and shared-client isolation - #4686

Merged
aheritier merged 4 commits into
mainfrom
test/claude5x-lifecycle-pricing
Oct 21, 2026
Merged

aheritier merged 4 commits into
mainfrom
test/claude5x-lifecycle-pricing

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

Part of #4654 (PR 1 of 3). Part of #4583 and epic #4580.

What

Three small tests pin Claude 5.x lifecycle behaviors that already work. Test-only: no production code changes, and no new test harness. Each test reuses what exists, per #4654's "test cost matters" rule.

Test Pins Runtime
TestEmbeddedSnapshotHaiku55ContextTier (pkg/modelsdev) Haiku 5.5's price tier in the embedded catalogue: base rates up to 100,000 prompt tokens, tier rates above (99,999 / 100,000 / 100,001), for anthropic, which also prices Vertex, and for amazon-bedrock global and us. (+10%) 0.03s
TestRoutedPricing/catalog_leaf_survives_reload (pkg/runtime) After a SQLite save and reload, a routed turn keeps the serving model's ID (not the router's) and its cost 0.02s
TestSharedClientConcurrentSessions (pkg/model/provider/anthropic) Two concurrent sessions on one client don't share system prompt, user text, attachments or tools; each is sent exactly once, as itself 0.01s, run 20× under -race

Each test was checked by deliberately breaking the behavior it pins and confirming it fails:

  • making the 100k threshold inclusive;
  • excluding Cost from the stored JSON;
  • leaking the system prompt through a package variable;
  • replaying one session's whole request to the other.

Not added, and why

The end-to-end runtime harness moves to PR 2, where compaction needs it.

References

Change Documentation
Haiku 5.5: "$0.10 / MTok for prompts up to 100,000 tokens", "$0.50 / MTok for prompts over 100,000 tokens"; cache reads and writes priced at matching rates Claude pricing
One client serves every conversation of a team load; per-conversation state belongs on the request AGENTS.md, "Client and Session Lifetimes"

Validation

  • task build passes.
  • task test passes. On a first run, e2e TestA2AServer_MultipleRequests failed (404 instead of 200); it passed 3/3 alone and on a full rerun. That's flaky under load, and this PR doesn't touch it.
  • golangci-lint v2.14.0 reports 0 issues, and go run ./lint . reports no offenses.
  • go test -race -shuffle=on ./pkg/runtime/ ./pkg/model/provider/anthropic/ ./pkg/modelsdev/ passes.
  • An independent review found 0 critical and 1 important issue: the concurrent test missed a whole-request substitution. It's fixed here.

Deferred minor findings:

  • The two requests are likely, not forced, to overlap.
  • TestRoutedPricing creates a temp dir for every case.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
Signed-off-by: Arnaud Héritier <aheritier@apache.org>
The leak check grouped requests by system prompt but never checked that
both conversations were seen, so a client replaying one session's whole
request to the other passed.

Signed-off-by: Arnaud Héritier <aheritier@apache.org>
@aheritier
aheritier requested a review from a team as a code owner October 21, 2026 11:10
Copilot AI balanced review requested due to automatic review settings October 21, 2026 11:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused test-only changes cover the stated regressions, with no unresolved blocking issues.

No open issues 🎉

What changed in this PR

Adds test-only coverage for Claude 5.x lifecycle behavior as the first part of #4654, reusing existing infrastructure without production changes.

Changes:

  • Pins Haiku 5.5 pricing at the 100,000-token boundary.
  • Checks that routed serving-model identity and cost survive SQLite reload.
  • Checks request isolation between concurrent sessions sharing one Anthropic client.
File Description
pkg/​runtime/​routing_pricing_test.go Verifies routed model and cost persistence.
pkg/​modelsdev/​snapshot_test.go Checks Anthropic and Bedrock pricing boundaries.
pkg/​model/​provider/​anthropic/​features_integration_test.go Checks shared-client session isolation.

Review effort: Balanced

@aheritier aheritier added area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/testing Test infrastructure, CI/CD, test runners, evaluation kind/test Test-only changes labels Oct 21, 2026
@aheritier
aheritier added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@aheritier
aheritier added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit a04fd30 Oct 21, 2026
17 checks passed
@aheritier
aheritier deleted the test/claude5x-lifecycle-pricing branch October 21, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/providers/anthropic For features/issues/fixes related to the usage of Anthropic models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/testing Test infrastructure, CI/CD, test runners, evaluation kind/test Test-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants