Skip to content

[python] Support creating primary-key BLOB tables and writing inline … - #10655

Open
zhuanshenbsj1 wants to merge 2 commits into
apache:masterfrom
zhuanshenbsj1:python-pk
Open

zhuanshenbsj1 wants to merge 2 commits into
apache:masterfrom
zhuanshenbsj1:python-pk

Conversation

@zhuanshenbsj1

@zhuanshenbsj1 zhuanshenbsj1 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

…BLOB fields on PK tables

Purpose

In Java Paimon (SchemaValidation.validateBlob), BLOB columns are supported on both append-only tables (primaryKeys.isEmpty(), requiring row-tracking. enabled = true and data-evolution.enabled = true) and primary-key tables (!primaryKeys.isEmpty(), with dedicated primary-key BLOB validation rules in validatePrimaryKeyBlob).

However, in pypaimon:
1. SchemaManager._validate_blob_fields unconditionally required row-tracking.enabled = true and data-evolution.enabled = true whenever a table had any
BLOB column, which prevented creating any primary-key BLOB table from Python (since row-tracking.enabled cannot be enabled on primary-key tables).
2. FileStoreWrite._create_data_writer blocked all primary-key tables containing any BLOB column, even when all BLOB columns were inline-stored (blob-descriptor-field or blob-view-field) and stored directly as BYTES inside normal Parquet/ORC/Avro data files without .blob sidecar files or rowIdtracking.

This PR aligns pypaimon schema validation with Java and enables creating primary-key BLOB tables, as well as writing and upserting inline BLOB fields
(blob-descriptor-field and blob-view-field) on primary-key tables in Python.

Tests

- `pytest paimon-python/pypaimon/tests/schema_manager_test.py` 
- `pytest paimon-python/pypaimon/tests/blob_table_test.py`

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python/Java parity and inline PK BLOB write/upsert use cases are useful. I found one schema-compatibility issue below. Verification: schema_manager_test.py and blob_table_test.py passed with PyArrow 19.0.1 (115 passed, 1 skipped, 36 subtests passed); additional format/bucket upsert and delete checks passed. Three Python-created partial-update BLOB configurations were independently rejected when loading the same persisted tables through Java FileStoreTableFactory.create.

Comment thread paimon-python/pypaimon/schema/schema_manager.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants