Repository navigation
Speed up offer lookups and consume them off the event loop - #4374
Merged
Merged
Conversation
The disk modifier deep-copied every cached offer before filtering on each get_offers() call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4373.
ComputeWithAllOffersCached.get_offers()deep-copied every cached offer to set its disk before filtering, and the lazy offers iterator was consumed bysorted()on the event loop, so a burst of offer lookups blocked the server.get_offer_disk_size_range()hook instead of a disk offer modifier, and copy only the offer, instance and resources instead of deep-copying: a lookup over GCP's 12.9k cached offers (44 regions) drops from 0.36–0.42 s to 0.02–0.03 s for GPU or default-disk requirements and to 0.09 s for broad requirements with an explicit disk range, with offers identical to master for the AWS, GCP, Azure, OCI, Nebius, RunPod and Verda catalogs.get_offers_by_requirements(): with 20 concurrent GCP lookups, the longest event-loop stall drops from 7–10 s, i.e. the whole burst, to 0.36 s for GPU lookups limited to 100 offers, 0.40 s for default-disk lookups and 2.1 s for broad explicit-disk lookups.Measured on an M1 Pro. Next step, outside this PR: consume offers in a separate single-worker pool. Concurrent lookups add no throughput under the GIL and take it away from the event loop; running them one at a time in the same 20-lookup bursts cut the longest stall to 0.18 s, 0.21 s and 0.49 s with the same total time.