Fix Iceberg partition and min/max pruning for timestamp_ns

[ianton-ru]

Frontport of #2239 onto antalya-26.8.

Identity partition pruning on Iceberg timestamp_ns compared Avro Int64 ticks at the wrong scale and dropped matching files. Min/max pruning also dropped files when Iceberg timestamp bounds were stored as nanoseconds. DateTime64(9) is now written as timestamp_ns (and timezone-aware DateTime64(9) as timestamptz_ns) on format version 3, and rejected on older format versions. Bounds whose magnitude is not above 1e18 fail open so mixed nanosecond and microsecond statistics are not guessed.

Related: #2239
Related: #2240

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix partition and min/max pruning for Iceberg tables by timestamp_ns type, fix writing DateTime64(9) as timestamp_ns.

Documentation entry for user-facing changes

Iceberg engine reference: DateTime64(9) writes map to timestamp_ns / timestamptz_ns on format version 3.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Made with Cursor

[github-actions]

Workflow [PR], commit [7555664]

[ianton-ru]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T16:02:13.715260Z	eab752b	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da866b680a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject unsupported nanosecond timestamps for Avro writes

When a v3 Iceberg table uses format=Avro, these returns advertise the column as timestamp_ns or timestamptz_ns, but AvroRowOutputFormat.cpp:214-219 serializes every scale-9 DateTime64 through the decimal fallback rather than as the long required by the Iceberg timestamp schema. An insert therefore produces a data file whose physical schema disagrees with the table metadata, so external Iceberg readers can reject or misdecode it; either implement the nanosecond timestamp encoding in the Avro writer or reject this type for Avro-backed Iceberg tables.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Remove the direct edit to generated reference docs

This paragraph is inside the page's AUTOGENERATED_START/AUTOGENERATED_END region, so it must not be changed directly. The corresponding source registration is already updated in registerStorageObjectStorage.cpp; keep that source change and let the dedicated documentation autogeneration workflow update this .mdx file.

AGENTS.md reference: docs/reference/AGENTS.md:L1-L3

Useful? React with 👍 / 👎.

[ianton-ru]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: eab752b954

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ianton-ru]

@blau-ai

[blau-ai]

CI triage for this PR (head eab752b)

Verdict: the red CI is almost entirely PR-caused — not flaky/infra. Every failing functional/integration test lands squarely in the timestamp_ns / DateTime64(9) code this PR adds or changes, and they reproduce 100% on reruns. Only the Grype scans, the Parser-memory-check error, and the long tail of DROPPED suites are non-PR (infra / cascade). So this needs code fixes before it can go green.

PR-caused (4 distinct failures)

1. 05385_iceberg_avro_timestamp_ns — new test, fails in both amd_debug, parallel and amd_asan_ubsan, distributed plan.
Reference expects the Iceberg data file to read back as Int64 (plain Avro long), but it comes back Decimal — the scale-9 column is still serialized as decimal bytes.
The long path in AvroSerializer::createSchemaWithSerializeFn gates on scale == 9 && column_mapper. On this IcebergLocal INSERT path the column_mapper isn't reaching the serializer, so the branch never fires. MultipleFileWriter builds the mapper and passes it via FormatFilterInfo, but it goes through getOutputFormatParallelIfPossible — check whether the parallel-formatting wrapper threads format_filter_info down to each inner AvroRowOutputFormat (AvroRowOutputFormat.cpp:760).
Suggested fix: mirror what the Parquet side of this very PR does — gate the scale-9 long schema on an explicit FormatSettings flag set in the Iceberg write path (like output_timestamp_adjusted_to_utc_from_timezone at MultipleFileWriter.cpp:247), instead of relying on column_mapper presence, which is fragile across the parallel path.

2. Integration DECIMAL_OVERFLOW (Code 407) — test_(writes|pruning)_datetime64_nanoseconds.py (new tests), ~6 cases across shards 2/8 and 5/8.
Thrown from getIdentityPartitionColumnValues → convertFieldToTypeOrThrow(partition_value, DateTime64(9)) at Utils.cpp:2124. Identity partition values for timestamp_ns arrive as raw Int64 nanosecond ticks (as your own wrapIntegerPartitionValuesAsDateTime64 comment notes); convertFieldToType then re-scales the Int64 by 1e9 → overflow.
Suggested fix: the Int64→DecimalField<DateTime64>(…, scale) wrapping you added in ManifestFilesPruning.cpp was only applied on the pruning path. Apply the same wrapping (or construct the DecimalField directly from the raw ticks) here in Utils.cpp::getIdentityPartitionColumnValues before convertFieldToTypeOrThrow.

3. Min/max pruning returns 0 files (assert 0 == 3) — test_minmax_pruning.py (12 cases, with_spark) and test_writes_statistics_by_minmax_pruning.py (6 cases, no_spark). These are pre-existing tests → a regression.
In IcebergFieldParseHelpers.cpp, deserializeDateTime64FromBinaryRepr for scale == 6 (Iceberg timestamp = DateTime64(6)) returns std::nullopt whenever |value| <= 1e18 (nanoseconds_lower_threshold). That window covers essentially all real microsecond timestamps (year 2024 ≈ 1.7e15 µs). deserializeFieldFromBinaryRepr now delegates all DateTime64 decoding to this function, so for normal µs stats both bounds come back null → no hyperrectangle is built → 0 files pruned for every ordinary Iceberg timestamp column.
Suggested fix: the "fail open" was meant for the genuinely ambiguous ns/µs overlap, but it currently fires for the common case. For scale == 6 with |value| <= 1e18, fall through and return the value as-is interpreted as microseconds (DecimalField<DateTime64>(DateTime64(value), 6)) instead of nullopt; reserve nullopt for only the narrow magnitudes that are actually ambiguous.

4. 04652_iceberg_mv_target_stale_schema_write — pre-existing stateless test, now fails (amd_debug, parallel).
Code: 36 … Iceberg type DateTime64(9) requires format version 3 or higher, got version 2 (BAD_ARGUMENTS). Your new "reject DateTime64(9) on format versions below 3" guard breaks this older test, which creates a DateTime64(9) IcebergLocal table on the default (v2) format.
Suggested fix: update 04652 to create that table with SETTINGS iceberg_format_version = 3 (or otherwise reconcile the test with the new guard).

Not PR-caused

• Grype Scan (keeper, server -alpine) — container CVE scans ("4 high/critical vulnerabilities" / "An error occurred"). This PR changes no Docker/dependency/packaging files, so these are base-image/infra findings, not from this change. (The non-alpine server image scan passed with 0.)
• Parser memory check — status ERROR, not a test assertion: No master clickhouse-examples artifact was found while resolving the master binary. Environment/baseline-artifact issue; safe to re-run.
• FinishCIReport + the many DROPPED suites (Stress, AST fuzzer, BuzzHouse, SQLLogic, Compatibility, etc.) — cascade from the integration-test failures ("Dropped due to previous failure"), not independent problems.

Bottom line

Fix items 1–4 (all in the Iceberg timestamp code) and re-run; the Grype/Parser-memory/dropped checks should clear on their own (Grype is a separate base-image concern unrelated to this PR). Happy to implement these — say the word and whether you want a direct commit to this branch or a separate blau/* PR.

Posted by @blau-ai. CI evidence from the praktika report for eab752b; I can't build/run ClickHouse in this container, so the fix directions above are from static reading and should be confirmed by CI.

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2610 @ eab752b

Verdict: every functional-test failure on this PR is PR-caused. The only red checks that are not about this PR's code are the Grype image scans and the Parser-memory-check (infra). Two of the PR-caused failures are regressions in pre-existing tests, which is the important part: this PR currently disables Iceberg min/max pruning for ordinary timestamp tables. Details and concrete fixes below.

🔴 PR-caused — min/max pruning regression (highest priority)

Pre-existing tests now failing, all combos, systematically (not flaky):

• test_storage_iceberg_with_spark/test_minmax_pruning.py::test_minmax_pruning — 12/12 params
• test_storage_iceberg_no_spark/test_writes_statistics_by_minmax_pruning.py::test_writes_statistics_by_minmax_pruning — 6/6 params

Both fail the same way — zero files pruned where 3 are expected:

E   assert 0 == 3
      ... WHERE ts <= timestamp('2024-03-20 14:00:00.000000')

Root cause — src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp, deserializeDateTime64FromBinaryRepr():

if (scale == 6)
{
    if (!magnitudeExceeds(unscaled, nanoseconds_lower_threshold))   // threshold = 1e18
        return std::nullopt;
    unscaled = nanosecondsToMicrosecondsForBound(unscaled, lower_bound);
}

A normal microsecond bound for a 2024 date is ~1.7e15, i.e. |ticks| <= 1e18, so magnitudeExceeds is false and the function returns std::nullopt. In IDataLakeMetadata.cpp:160 both left and right are then empty, so hyperrectangle is never built and min/max pruning is skipped for every ordinary timestamp/DateTime64(6) column — exactly the 0-vs-3 assertion.

Your own comment describes a three-band design (>1e18 ⇒ ns, convert; <=1e16 ⇒ normal micros; (1e16,1e18] ⇒ ambiguous, fail open), but the code only implements two bands and fails open for the entire <=1e18 range — the common case. The missing <=1e16 pass-through is the bug. Suggested fix (add a 1e16 constant and make it three-way):

// constexpr Int64 nanoseconds_ambiguous_threshold = 10'000'000'000'000'000LL; // 1e16
if (scale == 6)
{
    if (magnitudeExceeds(unscaled, nanoseconds_lower_threshold))            // > 1e18: stored as ns
        unscaled = nanosecondsToMicrosecondsForBound(unscaled, lower_bound);
    else if (magnitudeExceeds(unscaled, nanoseconds_ambiguous_threshold))  // (1e16, 1e18]: ambiguous
        return std::nullopt;                                               // fail open
    // else |ticks| <= 1e16: normal microseconds, use as-is
}
return DecimalField<DateTime64>(DateTime64(unscaled), scale);

(If you don't actually need the ambiguous-band fail-open, the minimal fix is to only convert when magnitudeExceeds(...) and otherwise return the value unchanged.)

🔴 PR-caused — DECIMAL_OVERFLOW in the new nanosecond identity-partition path

New tests added by this PR:

• test_storage_iceberg_no_spark/test_writes_datetime64_nanoseconds.py — ..._insert[timestamp_ns], [timestamptz_ns], test_writes_partition_pruning...
• test_storage_iceberg_no_spark/test_pruning_datetime64_nanoseconds.py, test_storage_iceberg_with_spark/test_pruning_datetime64_nanoseconds.py (partition-pruning variants)

The write/partition variants crash:

DB::Exception ... (DECIMAL_OVERFLOW)
  DecimalUtils::multiplyAdd<long,true>
  DecimalUtils::decimalFromComponents<DateTime64>
  convertFieldToType  (convertFieldToType.cpp:505)
  Iceberg::getIdentityPartitionColumnValues  (Utils.cpp:2124)

An identity-partition value for a timestamp_ns column is being re-scaled as if its components were seconds + fraction, then whole * 10^9 + frac overflows Int64 (a 2024 ns value is already ~1.7e18). This is a scale-handling bug in the new timestamp_ns partition-value path (getIdentityPartitionColumnValues / the field conversion feeding it) — please check that a ns partition value is fed to convertFieldToType already at scale 9 rather than being multiplied by 10^9 again.

🔴 PR-caused — new stateless test 05385_iceberg_avro_timestamp_ns (easy fix)

Failing in Stateless tests (amd_debug, parallel) and Stateless tests (amd_asan_ubsan, distributed plan, parallel, selected tests). stdout matches the reference exactly; the only failure cause is non-empty stderr:

Reason: having stderror:
<Warning> IcebergMetadata: Iceberg table schema was parsed using v2 specification, but it was impossible to parse it using v1 specification...

This is the benign v1→v2 fallback warning that any ClickHouse-written format_version>=2 table emits on read (IcebergMetadata.cpp:425); CI turns any stderr into a failure. Fix: silence server logs for this test the same way the sibling 04652 test does — add near the top, before sourcing shell_config.sh:

CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL="none"

🔴 PR-caused — regression in pre-existing 04652_iceberg_mv_target_stale_schema_write

Failing in Stateless tests (amd_debug, parallel):

Code: 36. DB::Exception: Iceberg type DateTime64(9) requires format version 3 or higher, got version 2. (BAD_ARGUMENTS)
(query: CREATE TABLE tgt_datetime64_9_... (c0 DateTime64(9)) ENGINE = IcebergLocal(..., 'Avro');)

On antalya-26.8, Utils.cpp mapped any DateTime64 → Iceberg timestamp (micros), so this test's datetime64_9 arm created the table at the default format version 2 and read back the truncated 2020-01-02 03:04:05.123456 (see its .reference). This PR now routes DateTime64(9) → timestamp_ns, which Utils.cpp:667 rejects below format version 3 — so the datetime64_9 arm's CREATE TABLE throws and aborts the whole batch. The test wasn't updated for the new behavior. Fix: in 04652, create the datetime64_9 target with SETTINGS iceberg_format_version = 3 and update the reference line to the full-precision 2020-01-02 03:04:05.123456789 (or drop the datetime64_9 arm, since 05385 now covers DateTime64(9)).

⚪ Not PR-caused (infra / image scans — safe to ignore for this change)

• GrypeScanKeeper / Grype Scan — "4 high/critical vulnerabilities" in the clickhouse-keeper base image; GrypeScanServer (-alpine) — "An error occurred" (scanner error). These scan the container images, not this PR's C++; unrelated to timestamp handling.
• Parser memory check — ERROR Resolve master binary: No master 'clickhouse-examples' artifact was found. Missing baseline artifact (infra); safe to re-run, not caused by the diff.
• FinishCIReport — aggregate gate; red only because of the failures above.

Bottom line

Not a flaky run — the two pre-existing min/max pruning suites (test_minmax_pruning, test_writes_statistics_by_minmax_pruning) going fully red is a real functional regression from the scale == 6 bound-deserialization change, and the new timestamp_ns write path overflows on identity partitions. I'd prioritize the deserializeDateTime64FromBinaryRepr fix and the getIdentityPartitionColumnValues overflow; the 05385 and 04652 test fixes are mechanical. Happy to push any of these to the PR branch if you want — just say which.

_{I analyzed CI only; correctness of any fix is validated by CI on the next push, not locally.}

[zvonand]

@blau-ai fix the PR-caused failing (and flaky) tests

[blau-ai]

Triaged the red checks on eab752b. I pushed fixes for the two failures I can attribute with confidence; two more are design conflicts inside the PR that I don't want to decide for you. Grype/Parser-memory/FinishCIReport are infra, not PR-caused (details at the bottom).

✅ Fixed — pushed to this branch (956936d)

1. DECIMAL_OVERFLOW (Code 407) reading any timestamp_ns/timestamptz_ns identity-partitioned table.
getIdentityPartitionColumnValues (Utils.cpp:2124) passed the raw Avro long nanosecond ticks straight into convertFieldToTypeOrThrow(..., DateTime64). For an integer source convertFieldToType treats the value as whole seconds and multiplies by the scale multiplier (decimalFromComponents → multiplyAdd), so 1.7e18 ns × 10^9 overflows. Even SELECT id FROM t crashes, because identity partition columns are materialised from the partition metadata on every read.

Fix: wrap the ticks into a DecimalField at the column scale directly — the same thing the PR already does on the pruning path (wrapIntegerPartitionValuesAsDateTime64). Regular (DecimalField) partition values are untouched.

This unblocks the Code 407 failures:

• test_writes_datetime64_nanoseconds.py::test_writes_identity_partitioned_datetime64_nanoseconds_insert[timestamp_ns|timestamptz_ns]
• test_writes_datetime64_nanoseconds.py::test_writes_partition_pruning_datetime64_nanoseconds[*]
• test_pruning_datetime64_nanoseconds.py::test_partition_pruning_datetime64_nanoseconds[*]
• test_pruning_datetime64_nanoseconds.py::test_spark_partition_pruning_datetime64_nanoseconds[*]

2. Stateless 05385_iceberg_avro_timestamp_ns — "having stderror".
The test output already matches the reference; it failed only because reading a ClickHouse-written format-version-3 table logs a benign IcebergMetadata: ... parsed using v2 specification ... writing engine violates Iceberg specification warning. That warning fires for every v3 read: parseTableSchema only short-circuits for format_version == 2, so v3 goes through the v1-first fallback, which fails (v3 metadata has no top-level schema) and warns. I suppressed server logs in the test (CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL="none"), matching the other Iceberg .sh tests (e.g. 04652).

Side note: that "your writing engine violates the Iceberg spec" warning is a false positive on ClickHouse's own v3 output and will spam logs for all real v3 reads. A cleaner root-cause fix is to let parseTableSchema use the v2 method for format_version >= 2 (not just == 2). I left it out of this commit because it touches the shared read path and belongs to you to decide — happy to add it if you want.

⚠️ Needs your call — design conflicts I didn't want to silently resolve

3. min/max pruning regressed for ordinary timestamp columns — test_minmax_pruning (12, with_spark) and test_writes_statistics_by_minmax_pruning (6, no_spark). Both are pre-existing tests (not in this PR's diff), both fail assert 0 == 3 on WHERE ts <= timestamp('2024-03-20 14:00:00.000000').

Root cause: deserializeDateTime64FromBinaryRepr returns std::nullopt for a scale-6 bound whose magnitude is <= 1e18. Normal µs timestamps (2024 ≈ 1.7e15) are all below that, so every ordinary Iceberg timestamp / DateTime64(6) column now loses its lower/upper bound and can't be min/max-pruned. In test_minmax_pruning the date <= case still prunes (line 112, Date column), only the ts <= case (line 116) returns 0.

The hard part: this collides with your own new tests. check_minmax_near_epoch_ns_bounds_on_timestamp_schema and check_minmax_overlap_band_ns_bounds_on_timestamp_schema deliberately feed near-epoch / 4.32e15 nanosecond bounds that land in the same magnitude range as legitimate microsecond timestamps and assert fail-open (pruned == 0). A pure magnitude heuristic can't tell "µs on a µs schema" from "ns on a µs schema" in that band, so you can't satisfy both sets at once. Directions, roughly:

• (a) Default scale-6 bounds to microseconds (restore pruning for the common case) and accept the OTEL ns-on-µs edge over-prunes — i.e. drop the near-epoch/overlap fail-open tests; or
• (b) Gate the ns-reinterpretation behind an explicit setting so ordinary tables keep pruning; or
• (c) Decide ns-vs-µs from schema/writer metadata instead of value magnitude.

I didn't touch this because any change breaks one side or the other and it's a correctness/product decision.

4. 04652_iceberg_mv_target_stale_schema_write (pre-existing, stateless) — broken by the new v3 requirement. Its datetime64_9 arm creates tgt_datetime64_9 (c0 DateTime64(9)) ENGINE = IcebergLocal(..., 'Avro') at the default format version (2). The new getIcebergType throws Iceberg type DateTime64(9) requires format version 3 or higher at CREATE TABLE, which aborts the batched setup query and cascades into UNKNOWN_TABLE for the rest of the arms. Its reference still expects the old lossy behaviour (datetime64_9: 2020-01-02 03:04:05.123456, written as µs). Pick one:

• (a) set SETTINGS iceberg_format_version = 3 for that arm and update the reference to full ns (.123456789); or
• (b) if DateTime64(9) on v1/v2 should keep writing as µs timestamp instead of throwing, relax getIcebergType and adjust the docs line that says v1/v2 reject scale 9.

Tell me which way you want 3 and 4 to go and I'll push the change.

Not PR-caused

• Grype Scan (keeper, server-alpine): image CVE scans, unrelated to this diff.
• Parser memory check: errored at Resolve master binary (infra), not a real test failure.
• FinishCIReport: just aggregates the above.

All of the above is validated by reasoning over the CI reports + code, not a local build — CI on this branch has to confirm fixes 1 and 2 (I can't run clickhouse-test from here).