Support AUTO and additional compression codecs for CSV/JSON scans

[osipovartem]

Summary

• Add AUTO header detection for CSV/JSON streams: gzip, bzip2, xz, zstd, and zlib Deflate; explicit Brotli and raw Deflate remain available.
• Filter AUTO directory listings by supported CSV/JSON suffixes while preserving explicit file paths.
• Preserve byte-range parallelism for large plain CSV: probe at most 32 file headers concurrently (6 bytes each), switching to UNCOMPRESSED only when every file is plain. Mixed codecs and probe failures retain AUTO.

Verification

• DataFusion codec/listing tests pass; one-file and two-file large plain CSV tests execute and count 9M and 1.2M rows respectively with split scan groups.
• cargo fmt --all --check and focused cargo clippy --all-targets -- -D warnings pass.
• cargo check -p datafusion --no-default-features --features nested_expressions,crypto_expressions,datetime_expressions,encoding_expressions,regex_expressions,string_expressions,unicode_expressions,parquet,recursive_protection,sql --lib passes.
• Independent read-only review approved correctness, performance, API compatibility, and tests.

Bounds

Plans with >32 files do not pay for header probes; file-level parallelism remains. Mixed-codec scans stay in AUTO and cannot use byte-range splitting. Base branch matches the DataFusion revision pinned by Rustice, so this PR contains only compression changes.