Skip to content

[core] Support copying stream scan state in Python bindings - #1187

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/stream-scan-state-copy
Oct 21, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/stream-scan-state-copy

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Support independent copies of streaming scan progress and configuration, including the standard Python copy.copy(scan) protocol.

The PyPaimon Native bridge decodes a Rust plan into its public split objects. When that conversion fails, restoring only the next snapshot ID leaves the Rust watermark and other scan state advanced. A later empty poll can then expose the failed frame's watermark. Copying the complete state before conversion allows the bridge to restore the original scan without changing Java-compatible restore(next_snapshot_id) semantics, filtering watermark values, or performing additional metadata reads.

Changes

  • Implement Clone for Core StreamTableScan, preserving independent cursor, watermark, consumer configuration, restored state and missing-snapshot polling counters.
  • Implement Python StreamTableScan.__copy__ and update the stub. The copy owns separate bucket-filter callback state and Python references, with a rebuilt Core callback closure. Each wrapper traverses exactly its own references for GC.
  • Reuse the existing filter installation logic. Callback replacement and clearing release Python references outside the state lock, and callbacks run without holding that lock.
  • Add behavior tests for independent progress, configuration, consumer acknowledgements, callback ownership, exception identity and GC cycles.

No new runtime dependencies or changes to checkpoint/restore/consumer acknowledgement semantics. This supports the Native execution alignment in apache/paimon#10712.

Verification

  • Core clone regression: 1 passed, using a real memory table and independent checkpoint/watermark/filter/consumer operations.
  • Rust 1.98.0 cargo check --locked -p pypaimon_rust --tests passed.
  • Rust 1.98.0 Clippy with -D warnings passed for Python bindings/tests and the Core stream test target; workspace formatting and diff checks passed.
  • The focused companion Python stream/auth suite passed 358 tests, 11 skipped, including exact rollback cases for waiting polls, irrelevant snapshots and valid empty snapshots with the same watermark.
  • The embedded Python Rust unit test stream_scan_copy_keeps_callback_ownership_and_progress_independent: 1 passed, covering both callback ownership modes, independent cursors and collection of both wrappers' reference cycles. It runs as a Rust test binary without producing or installing a Python extension.
  • Independent review checked callback reference ownership, GC traversal, reentry and full scan-state restoration.
  • No Python extension was rebuilt or installed locally. The six new binding integration tests require the PR's extension in CI; the existing local extension predates this API.

Main-branch CI repairs

Also repair the two failures reported on main at e4ec3a45:

  • Python binding integration: the sole failing test still expected four selected rows despite the configured reader limit of two. Align the expectation with current Core/Java read semantics. Separately check that scan planning retains the entire position slice, then check that the configured reader enforces its limit. Parameterize no-limit/limit cases with row tracking enabled and disabled, and add the equivalent real-Parquet Core regression. Correct the stale with_limit rustdoc.
  • Ubuntu unit: Core tests succeeded; the runner failed with System.IO.IOException: No space left on device during the subsequent REST-server test step. Use line-table debug information and disable incremental caches for the unit job, retaining backtrace line numbers while reducing accumulated build artifacts. Preserve every test command, OS matrix entry and the existing Windows concurrency limit.

Verification: the new Core regression passed all four combinations; its Rust 1.98 Clippy target and workspace formatting passed. Both no-limit binding cases passed using the existing installed extension. That older local extension predates reader-limit enforcement, so the updated limited binding cases rely on the current-Core regression and PR CI; the extension was not rebuilt or installed. YAML validation confirmed the unit environment and unchanged test commands/platforms. Hosted-runner disk usage is validated by the new CI run rather than estimated locally.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e602ec5 into apache:main Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants