Skip to content

Allow explicit partition values for prelisted files - #135

Merged
osipovartem merged 1 commit into
df55-copy-compression-autofrom
df55-file-origin-provenance
Oct 10, 2026
Merged

osipovartem merged 1 commit into
df55-copy-compression-autofrom
df55-file-origin-provenance

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

Allow a prelisted ListingTable to expose caller-supplied partition values, such as a numeric source-file ID, without listing or HEAD requests. Validate count, type and non-nullness when constructing the table. Apply partition predicates after path deduplication and before file limits so pushed-down exact filters remain correct.

The normal listing path and the no-filter prelisted scan path retain their existing behavior. This is a DataFusion building block for per-file COPY INTO accounting in Rustice; it does not itself change COPY result rows.

Verification

  • cargo +1.97.0 fmt --all --check
  • cargo +1.97.0 test --profile ci -p datafusion --test core_integration datasource::object_store_access:: (27 passed; CSV, JSON, Parquet, filtering and duplicate-path cases)
  • cargo +1.97.0 test --profile ci -p datafusion-catalog-listing --lib (20 passed)
  • cargo +1.97.0 clippy --profile ci -p datafusion-catalog-listing -p datafusion --tests -- -D warnings

Independent read-only review: approved, no blocking findings remaining.

@osipovartem
osipovartem merged commit f86445b into df55-copy-compression-auto Oct 10, 2026
7 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant