Skip to content

Create CSV record handlers per opened file - #137

Merged
osipovartem merged 1 commit into
df55-auto-zlib-header-checkfrom
df55-csv-file-handler-factory
Oct 10, 2026
Merged

osipovartem merged 1 commit into
df55-auto-zlib-header-checkfrom
df55-csv-file-handler-factory

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

  • Supports Embucket/rustice#594. This PR adds the reusable CSV API; it does not close Rustice's COPY issue by itself.

Rationale for this change

A shared CSV record handler cannot attribute decoded or malformed records to their source file in a multi-file scan. Consumers needing precise file-local diagnostics must currently build one scan per file. In a 128-file local COPY benchmark, that approach regressed p50 latency from 33.4 ms to 370.8 ms. The CSV opener already knows the object's path when it creates each reader, so an opt-in file-scoped handler avoids the per-file scan graph.

What changes are included in this PR?

  • Add CsvRecordErrorHandlerFactory, selected once per opened object-store path by CsvOpener.
  • Expose it through CsvFormat and CsvSource; setting either a shared handler or factory uses last-call-wins semantics.
  • Keep the default path unchanged. Factory scans disable byte-range repartitioning and remain non-serializable, like shared-handler scans.
  • Test successful and malformed records from two files using distinct path-scoped handlers.

Are these changes tested?

  • cargo +1.97.0 fmt --all -- --check
  • cargo +1.97.0 clippy --all-targets --all-features -- -D warnings
  • Full extended DataFusion workspace tests with avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption; passed after initializing pinned test-data submodules.
  • datafusion-datasource-csv unit tests: 14/14 passed.
  • Independent read-only review: APPROVED, no blocking findings.

Are there any user-facing changes?

Additive public API only. Existing shared-handler and default CSV behavior are unchanged. No breaking API change.

@osipovartem
osipovartem merged commit 78abddc into df55-auto-zlib-header-check Oct 10, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant