Skip to content

feat(core): protect local metadata and global index cache blocks - #1184

Merged
JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/local-cache-priority-pools-20261020
Oct 20, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/local-cache-priority-pools-20261020

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Keep one local-cache size budget while protecting metadata and global-index blocks in a separate LRU. Either priority class can borrow unused capacity.
  • Add local-cache.high-priority-pool-ratio (default 0.25; 0 restores a single LRU). Apply the policy to memory and disk caches, including disk recovery, without changing the on-disk block format.
  • Cover eviction, disabled priority, disk restart, configuration, and real reader behavior.

Validation

  • cargo fmt --all -- --check: passed
  • cargo test -p paimon --lib cache: 129 passed (storage-fs,storage-memory)
  • cargo clippy -p paimon --lib --tests: passed with two pre-existing storage.rs warnings suppressed under this feature set
  • Full lib run: 4,322 passed, 1 failed, 7 ignored. The failure was empty_target_does_not_bypass_mutation_authorization in a parallel run; it passed when rerun alone. No local-cache code is on that test path.

No production OSS/QPS benchmark was run. This PR targets current main; the unmerged cross-process shared cache in #1175 will need a separate integration with the same policy.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 20, 2026 11:31

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 2073ce5 into apache:main Oct 20, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants