Skip to content

Antalya 26.8, Backport of 115686: Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB] - #2619

Open
ianton-ru wants to merge 1 commit into
antalya-26.8from
backport/antalya-26.8/115686
Open

ianton-ru wants to merge 1 commit into
antalya-26.8from
backport/antalya-26.8/115686

Conversation

@ianton-ru

Copy link
Copy Markdown

Backport of ClickHouse#115686, squashed into one commit.

The v3 Parquet reader always read a fixed 64 KiB tail to fetch FileMetaData. That misses the footer of a wide file and costs a second round trip, while a read between 128 KiB and 2 MiB has almost the same latency on object storage. The initial read is now 1% of the file size, clamped to [128 KiB, 2 MiB]. input_format_parquet_footer_read_size overrides that with a fixed size; 0 keeps the adaptive size.

On this branch the setting is recorded under 26.8.27.20001.altinityantalya, so compatibility with 26.8 restores the previous fixed 64 KiB read. The bucket splitter reads the footer through the same v3 path, and the Delta column-pruning test pins the read size so its byte counts stay a measure of pruning.

05385_parquet_footer_read_size passed locally against clickhouse local.

Related: ClickHouse#115686
Related: #2503

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

More optimistic Parquet footer-size estimate instead of a fixed 64 KiB, fetching the footer in one read instead of two and reducing footer-read latency. input_format_parquet_footer_read_size overrides the size; compatibility with 26.8 keeps the old 64 KiB read.

Documentation entry for user-facing changes

No user-facing documentation change. The setting description is on input_format_parquet_footer_read_size.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

The v3 reader always read a fixed 64 KiB tail, which misses the footer of a wide file and costs a second round trip. Size that read to 1% of the file, clamped to [128 KiB, 2 MiB], and add `input_format_parquet_footer_read_size` to override it. `compatibility` with 26.8 keeps the old 64 KiB read.

https: //github.com/ClickHouse/pull/115686
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown

Workflow [PR], commit [9af546a]

@ianton-ru ianton-ru added antalya backport Backport antalya-26.8 Session label (releasy session config) labels Oct 20, 2026
@ianton-ru ianton-ru changed the title Antalya 26.8 - Backport of 115686 - Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB] Antalya 26.8. Backport of 115686: Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB] Oct 20, 2026
@ianton-ru ianton-ru changed the title Antalya 26.8. Backport of 115686: Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB] Antalya 26.8, Backport of 115686: Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB] Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.8 Session label (releasy session config) backport Backport

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant