[XiaoHongbo-Hope]

Summary

Add opt-in local-cache.shared=true for cooperating reader processes on one
node. The existing process-local memory/disk cache remains the default.

This shares raw immutable file blocks, not parsed Parquet metadata or decoded
Arrow values. It can be enabled by native SDK callers passing catalog options:

local-cache.enabled=true
local-cache.dir=/node-local/paimon-cache
local-cache.shared=true
local-cache.max-size=20 GiB
local-cache.whitelist=meta,global-index,data

Implementation

• Separate versioned directory, same-UID local filesystem only, with explicit
  matching capacities for participating processes.
• Standard-library file locks coordinate access. Shared directory locks allow
  concurrent cache hits; exclusive locks serialize publication, accounting,
  eviction, and invalidation. Striped download locks coalesce concurrent misses,
  including full/range reads. Read/publication contention with an exclusive
  cache operation is bounded and falls back to the original source.
• Atomic block publication reuses the existing checksummed cache encoding.
• Persistent recency and batched eviction bound encoded block capacity across
  processes; interrupted accounting is recovered before another publication.
• A shared generation fences old readers after invalidation. Rename establishes
  this fence before changing storage and holds the exclusive gate across the
  storage operation, so an uncertain rename result cannot expose stale blocks.
  Invalidation is intentionally directory-wide, even for one logical path.
• Missing or malformed generations are replaced at startup after unknown blocks
  are removed; process-local file-size caching is bypassed in shared mode.
• Clean worker startup reuses valid persisted accounting without scanning blocks;
  dirty, missing, malformed, or over-capacity accounting triggers recovery.

Scope / draft review questions

This is a draft for design review, not a production rollout recommendation.

• Large-cache, multi-worker throughput and real OSS request savings still need
  measurement. Hits run concurrently, but publication and invalidation still use
  the directory-wide exclusive gate.
• Directory-wide invalidation is conservative and can hurt write-heavy workloads.
• Invalidation lock timeout / filesystem errors still propagate through delete,
  writer flush/close, and streaming shutdown. Storage may already have changed
  when those errors are reported; callers must not reuse paths until invalidation
  succeeds. Rename instead waits for a mandatory pre-mutation fence; cleanup after
  generation rotation is recoverable and does not make a successful rename appear
  to fail.
• Participants must initialize catalogs/readers inside workers, not reuse
  inherited live runtimes/caches after fork.
• The existing parsed Parquet metadata cache ([rust] Cache parsed Parquet metadata across readers #929) has a separate inherited-lock
  hazard after fork. This feature does not claim to fix that issue.
• NFS / cross-node sharing is out of scope. Control files and one temporary
  in-flight block are outside the encoded-block budget.

Validation

Rust 1.94, macOS:

The cache-filtered suite passes 133 tests on the proposed source, including
the separate-process tests below. Formatting and whitespace checks pass.

cargo +1.94.0 test -p paimon --lib cache \
  --no-default-features --features storage-fs,storage-memory
cargo +1.94.0 fmt --all -- --check
git diff --check

Tests cover separate executable processes sharing one source download,
CachedFileReader range/full reads, process termination releasing locks,
cross-process invalidation fencing stale readers, capacity mismatch, corruption,
and interrupted accounting recovery. Added regressions cover concurrent cache
hits under shared directory locks, malformed-generation repair, and rename
waiting for its pre-mutation fence without exposing a stale destination. Existing
coverage also includes invalidation lock timeout and filesystem failure, FileIO
file/directory deletion followed by retry and recreation, repeated streaming
shutdown failures, and eight independent worker startups on 2,000 blocks
asserting zero recovery scans. This is a functional test, not a production-scale
startup throughput benchmark. No production tables or customer programs are
modified.

[JingsongLi]

+1