Repository navigation
Conversation
The v3 reader always read a fixed 64 KiB tail, which misses the footer of a wide file and costs a second round trip. Size that read to 1% of the file, clamped to [128 KiB, 2 MiB], and add `input_format_parquet_footer_read_size` to override it. `compatibility` with 26.8 keeps the old 64 KiB read. https: //github.com/ClickHouse/pull/115686 Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of ClickHouse#115686, squashed into one commit.
The v3
Parquetreader always read a fixed 64 KiB tail to fetchFileMetaData. That misses the footer of a wide file and costs a second round trip, while a read between 128 KiB and 2 MiB has almost the same latency on object storage. The initial read is now 1% of the file size, clamped to[128 KiB, 2 MiB].input_format_parquet_footer_read_sizeoverrides that with a fixed size;0keeps the adaptive size.On this branch the setting is recorded under
26.8.27.20001.altinityantalya, socompatibilitywith26.8restores the previous fixed 64 KiB read. The bucket splitter reads the footer through the same v3 path, and the Delta column-pruning test pins the read size so its byte counts stay a measure of pruning.05385_parquet_footer_read_sizepassed locally againstclickhouse local.Related: ClickHouse#115686
Related: #2503
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
More optimistic
Parquetfooter-size estimate instead of a fixed 64 KiB, fetching the footer in one read instead of two and reducing footer-read latency.input_format_parquet_footer_read_sizeoverrides the size;compatibilitywith26.8keeps the old 64 KiB read.Documentation entry for user-facing changes
No user-facing documentation change. The setting description is on
input_format_parquet_footer_read_size.CI/CD Options
Exclude tests:
Regression jobs to run: