Repository navigation
fix(runtime): auto-refresh stale exhausted quota and back off probing - #312
Merged
Merged
Conversation
…ocked routing A stale Exceeded=true quota snapshot kept an account out of routing until an operator manually refreshed it: nothing re-fetched quota in the background, and a pool with every account quota-exhausted produced no routing hint, so clients saw a generic "no accounts available" instead of a quota error. - maintenance loop re-fetches quota every 5 minutes for accounts whose cached snapshot still reports Exceeded and is older than 5 minutes; healthy accounts are never touched (they are not in scope), disabled accounts are already absent from the pool, and dead accounts are skipped - pick() returns 429 insufficient_quota (cooldown to local midnight) when every account that could serve the model is blocked only by exhausted quota, instead of "no accounts available" - routeBaseMatches split into quota gate + routeReadyAndScoped so the quota-blocked check reuses the scoping conditions without duplication - unit tests for both; changelog fragment (bilingual)
The stale-quota sweep re-fetched every Exceeded account on a fixed five minute cadence, so an account that stayed empty for days cost ~288 quota calls a day for no routing change. Provider balances reset on a daily cadence, so that polling rate buys nothing. Each account now carries a consecutive-exhausted level that selects the next probe delay: 5m, 5m, 5m, 15m, 30m, 1h, 3h, 6h, 12h, then 24h. The first three probes stay fast so a transient false Exceeded or an out-of-band top-up still recovers within minutes. The level resets to the fast end as soon as a snapshot reports a non-zero balance, and a failed probe keeps its level so a transient outage is not escalated into an hours-long wait. Level state is pruned to the current pool so it cannot grow without bound. Healthy accounts are still never touched by the sweep.
One bilingual fragment per PR, per changelog/unreleased/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A stale
Exceeded=truequota snapshot kept an account out of routing until an operator manually refreshed it, and an all-exhausted pool produced only a generic "no accounts available".Exceededand is stale; healthy accounts are never touched.pick()returns429 insufficient_quotawhen every account that could serve the model is blocked only by exhausted quota.routeBaseMatchessplit into the quota gate +routeReadyAndScopedso the quota-blocked check reuses scoping without duplication.Tests
go build ./...,go vet ./...,go test ./...,git diff --check