Repository navigation
Rollup writes now lock per row, through 64 lock stripes per partition - #4
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rollup writes now lock per row, through 64 lock stripes per partition, instead of once per tree. Ingestion is about 2.6× faster without rollup indexes and 3.1× faster with them. The full suite passes, 870 of 870.
Changes
RollupWriterhashes each key to one of 64 stripes, each with its own lock and its own pending replacement. Workers update different rows of a partition in parallel; the same row is still updated one writer at a time.TryAtomicAddOrUpdate.Validation
WritesOfRowsInOtherLockStripesDoNotWait: a copy hook holds one row's lock while a row in another stripe is written. It failed on the per-tree lock, where the second write waited.ConcurrentWritersOfTheSameRowsAddEveryDelta: 16 writers make 16,000 updates over 50 overlapping rows, and every delta must be counted exactly.Measured (playground, 1M events over 365 days, 20 workers, two alternating rounds; one stripe reproduces the old lock):
Compaction after ingestion takes 1–3 s longer, because more data arrives sooner. Rollup indexes now cost about 29% of throughput instead of 40%.