Skip to content

[core] Support scheduled execution of snapshot-scoped search plans - #1191

Open
JingsongLi wants to merge 2 commits into
apache:mainfrom
JingsongLi:codex/search-execution-scheduler
Open

JingsongLi wants to merge 2 commits into
apache:mainfrom
JingsongLi:codex/search-execution-scheduler

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Paired PyPaimon integration: apache/paimon#10724.

Allow PyPaimon to assemble ordinary search source plans and use Ray for placement while Rust Core owns query execution and global coordination.

Changes

  • Make VectorScanPlan and FullTextScanPlan snapshot-and-split data with public constructors, synchronous source access and validated serde restoration. Plans carry no live table, SDK handle, query configuration or fingerprint.
  • Execute explicitly supplied sources without discovering additional indexes or uncovered raw ranges. Readers own the schema, query, options, filters, authorization and FileIO, and resolve data-file/deletion-vector metadata at the captured snapshot.
  • Preserve global ANN candidate selection before refinement, one raw BM25 corpus and hybrid rank fusion. Keep task batching bounded and share repeated immutable inputs between workers.
  • Keep shard task transport out of the default public Core API. Repository bindings use a hidden internal-api bridge; the Python SDK exposes public Plan constructors and Read methods, with private scheduler callbacks.
  • Support transported plans, reader-added filters, captured empty plans, primary-key source payloads and schema-only field renames. Capture full snapshots for decoded Java bucket splits without changing the C ABI.
  • Derive indexed partition masks from pinned data-file row ranges. Keep full predicates on raw sources and intersect partition, scalar and caller masks before ANN Top-K.

Validation

  • Rust 1.98.0 Core vector tests: 79 passed; full-text tests: 40 passed; hybrid tests: 22 passed.
  • Primary-key vector unit tests: 31 passed; Java bucket-split integration tests: 7 passed.
  • Python binding constructor/metadata codec tests: 3 passed; scheduler callback tests: 3 passed. These exercise real Python binding classes, malformed source payloads, concurrency, original cancellation errors and generator cleanup.
  • C bridge reading the Java bucket-split fixture: 1 passed.
  • Strict Clippy for default Core, full-text Core test targets and Python binding production/test targets passed with Rust 1.98.0 and -D warnings; formatting passed.

The Native extension is not rebuilt or installed into the PyPaimon test environment. There is no storage-format change or distributed primary-key search extension.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant