Repository navigation
[core] Support copying stream scan state in Python bindings - #1187
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Support independent copies of streaming scan progress and configuration, including the standard Python
copy.copy(scan)protocol.The PyPaimon Native bridge decodes a Rust plan into its public split objects. When that conversion fails, restoring only the next snapshot ID leaves the Rust watermark and other scan state advanced. A later empty poll can then expose the failed frame's watermark. Copying the complete state before conversion allows the bridge to restore the original scan without changing Java-compatible
restore(next_snapshot_id)semantics, filtering watermark values, or performing additional metadata reads.Changes
Clonefor CoreStreamTableScan, preserving independent cursor, watermark, consumer configuration, restored state and missing-snapshot polling counters.StreamTableScan.__copy__and update the stub. The copy owns separate bucket-filter callback state and Python references, with a rebuilt Core callback closure. Each wrapper traverses exactly its own references for GC.No new runtime dependencies or changes to checkpoint/restore/consumer acknowledgement semantics. This supports the Native execution alignment in apache/paimon#10712.
Verification
cargo check --locked -p pypaimon_rust --testspassed.-D warningspassed for Python bindings/tests and the Core stream test target; workspace formatting and diff checks passed.stream_scan_copy_keeps_callback_ownership_and_progress_independent: 1 passed, covering both callback ownership modes, independent cursors and collection of both wrappers' reference cycles. It runs as a Rust test binary without producing or installing a Python extension.Main-branch CI repairs
Also repair the two failures reported on main at
e4ec3a45:with_limitrustdoc.System.IO.IOException: No space left on deviceduring the subsequent REST-server test step. Use line-table debug information and disable incremental caches for the unit job, retaining backtrace line numbers while reducing accumulated build artifacts. Preserve every test command, OS matrix entry and the existing Windows concurrency limit.Verification: the new Core regression passed all four combinations; its Rust 1.98 Clippy target and workspace formatting passed. Both no-limit binding cases passed using the existing installed extension. That older local extension predates reader-limit enforcement, so the updated limited binding cases rely on the current-Core regression and PR CI; the extension was not rebuilt or installed. YAML validation confirmed the unit environment and unchanged test commands/platforms. Hosted-runner disk usage is validated by the new CI run rather than estimated locally.