Skip to content

Avoid false AUTO zlib detection on printable CSV prefixes - #136

Merged
osipovartem merged 3 commits into
df55-copy-compression-autofrom
df55-auto-zlib-header-check
Oct 10, 2026
Merged

osipovartem merged 3 commits into
df55-copy-compression-autofrom
df55-auto-zlib-header-check

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

No issue filed; this fixes a reproducible COPY INTO ... COMPRESSION=AUTO failure on ordinary CSV input.

Rationale for this change

An uncompressed CSV beginning with 80, passes the two-byte RFC 1950 zlib-header check. AUTO then sends it to the DEFLATE decoder and fails the load. A focused live Snowflake COPY of the same plain input with COMPRESSION=AUTO succeeds.

What changes are included in this PR?

  • Probe up to 64 bytes only for zlib-header candidates. Printable text takes precedence, and otherwise a bounded decompression attempt rejects invalid zlib prefixes.
  • Use the same prefix check in CSV repartition planning. Preserve the six-byte fast path for non-candidates in the synchronous reader and preserve explicit DEFLATE.
  • Add regression tests for plain 80, CSV, printable zlib-looking prefixes, genuine zlib, truncated input, and the six-byte noncandidate read path.

Are these changes tested?

  • datafusion-datasource: 185/185; datafusion-datasource-csv: 13/13, with compression enabled.
  • Full cargo clippy --all-targets --all-features -- -D warnings and cargo fmt --all -- --check pass locally.
  • Rustice's targeted SQLLogicTest and a local many-file COPY benchmark passed on the preceding revision; final pinned-SHA integration is running separately.
  • Independent read-only review approved the final diff. The full extended DataFusion workspace test suite was not completed because the WSL host disk has limited free space; affected-crate tests and full-workspace lint were completed.

Are there any user-facing changes?

COMPRESSION=AUTO no longer misclassifies this class of plain CSV as zlib. A finite prefix cannot disambiguate every stream; users can still specify DEFLATE or NONE for ambiguous files. No public API change.

@osipovartem
osipovartem merged commit 74db011 into df55-copy-compression-auto Oct 10, 2026
12 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant