Skip to content

[Python] Unify bound execution protocols and writer configuration lifecycle - #10717

Merged
JingsongLi merged 6 commits into
apache:masterfrom
JingsongLi:codex/execution-protocol-unification
Oct 21, 2026
Merged

JingsongLi merged 6 commits into
apache:masterfrom
JingsongLi:codex/execution-protocol-unification

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Continue aligning Python and Rust execution APIs while preserving public PyPaimon contracts. This follows the merged #10712.

Depends on apache/paimon-rust#1188 for captured append writer configuration and the private row transport query. Native CI continues to build Rust main; this PR remains draft until that dependency is merged.

Changes

  • Replace unbound executor mixin calls with explicit bound Python/Native scan and read executors. Both backends use the same actual operations, while public hooks continue to observe live configuration.
  • Bind Python writer, row-ID/predicate update, delete, upsert, MERGE and shard executors to their public contexts. Prepared upsert/MERGE requests use matching execution entry points on either backend. No generic proxies, compatibility aliases or cross-backend imports are added.
  • Give writer executors one explicit prepare_commit(identifier) boundary. Public batch facades accept no arguments and retain the one-time guard; public stream facades require a signed 64-bit identifier. Native facades enforce the same signatures before preparation, including after an explicit switch to Python.
  • Align append reconfiguration with current Java: each partition/bucket captures its columns and Blob consumer, retaining them across prepares. Future buckets use the new configuration. Check every target layout before staging any group.
  • Keep Native row conversion thin: ask Core for destination columns using routing metadata only, retain the captured order and avoid opening unused Blob inputs. NOT NULL/event validation precedes this query. Core owns routing, schema history and physical preflight.
  • Delete the acceptance package and its tests, and remove its test/distribution configuration.
  • Remove the Python benchmark package, its ACT/RoboMIND tests and documentation, the benchmark-only act extra, pytest input option and packaged experiment resource.
  • Move deletion-vector bitmap codecs, file loading and row filtering to execution/python/deletionvectors, keeping shared DeletionFile / DeletionVectorMeta metadata in their domain modules. Update all callers and verify distribution contents.
  • Separate array cast execution and executable-cast probes into execution/python/casting. Put shared Arrow read schemas and logical cast rules in schema, shared numeric/calendar/metadata formatters in utils, and private Python commit partition-spec parsing in its executor. Native read/commit adapters do not import Python cast implementations.
  • Complete the remaining corresponding backend placement: move primary-key interval grouping, Blob View lookup, physical MAP/VARIANT codecs, dynamic-bucket HASH publication and MERGE condition/action execution into execution/python. Keep shared metadata/schema/value helpers, SQL parsing/validation, callable input adapters and cross-backend bucket input routing in their domain modules. Local and Ray MERGE use the same Python action factories.
  • Remove the remaining benchmark package path references from the packaging checks. Add fresh-process tests that import every Native adapter with Python execution and SDK initialization blocked, and restore a real HASH index through the shared input assigner with Python execution blocked.
  • Audit common against actual backend callers. Move external-file placement strategies into Python write execution and MAP BLOB physical key serializers into the Python backend. Split authorization JSON Arrow filtering/masking/CAST execution from shared field-reference extraction. Preserve public authorization factory signatures and shared path-factory configuration/existing-file lookup; load Python implementations only on explicit execution. URI/FileIO contracts, catalog-owned caches, generic integer codecs and shared predicate/output helpers remain outside the backends.
  • Document execution contracts and writer lifecycle. Update existing regression expectations for Java-style reconfiguration and propagation of prefetch errors.

Verification

  • Common/execution follow-up: 215 storage/Blob regressions passed, 117 Native isolation/protocol regressions passed, and 354 authorization/temporal regressions passed (168 Native cases skipped because the local SDK was not rebuilt). All 9 packaging tests pass and 22,363 current tests collect. Fresh-process checks block both execution backends during public authorization validation and block Python execution during Native existing-file path resolution with external placement configured.
  • Independent cross-review confirms both whole-module relocations and 35 authorization execution functions preserve their AST bodies, while shared date-transform field dependencies remain unchanged.
  • Final placement review: two independent cross-reviews found no further actionable issues. AST comparisons confirm moved functions/classes preserve their bodies; internal imports and distribution paths are updated.
  • Read/codecs: 290 tests passed (177 subtests; 4 local-SDK-dependent Native cases deselected). Blob View integration: 11 passed.
  • Boundary/Blob regressions: 207 passed; Native import/read/write/update protocol isolation: 117 passed. Local MERGE regressions: 48 passed, 16 optional cases skipped; two actual Ray MERGE regressions passed.
  • All 22,362 current Python tests collect successfully; all 9 sdist/wheel packaging tests pass. The source tree has no remaining benchmark package path references.
  • Cast relocation: 559 focused tests passed, 100 Native/optional cases skipped; 9 packaging tests passed. Fresh-process Native schema and commit-path adapters reject Pure imports, and all 22,357 tests collect successfully. Moved cast, calendar, schema and partition function bodies retain their prior semantics.
  • Benchmark removal: all 22,357 remaining Python tests collect successfully; the 9 packaging tests verify wheel/sdist contents and updated extras. Documentation sidebar syntax validation passes.
  • Deletion-vector relocation: 331 passed, 58 skipped; actual wheel/sdist contents test passes.
  • Focused read/write/update execution, lifecycle, public-hook, prepare-identifier, backend import-isolation and reconfiguration tests.
  • Native transport regressions use mock SDK handles; actual destination validation, routing, prepare/readback and callback GC are covered by the companion Rust tests.
  • 9 actual sdist/wheel packaging tests pass.
  • Full package Flake8, changed-file Python 3.6 grammar check, license headers and git diff --check pass.
  • Independent API/boundary reviews confirm existing public method arguments/defaults, class paths, hooks and backend isolation.

The local installed SDK predates this change. It was not rebuilt or installed, and no Native version probing/fallback or CI dependency override was added.

@JingsongLi
JingsongLi marked this pull request as ready for review October 21, 2026 11:17

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e008313 into apache:master Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants