Fix iceberg partition pruning by nanoseconds

[ianton-ru]

Solved #2240

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix partition and min/max pruning for Iceberg tables by timestamp_ns type, fix writing DateTime(9) as timestamp_ns.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [f884dae]

[ianton-ru]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 381b43d5e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Skip pruning instead of guessing nanosecond bounds

When an Iceberg timestamp manifest contains a magnitude above this threshold, the schema still identifies the value as microseconds; magnitude alone cannot prove that the writer actually used nanoseconds. Dividing the value and then trusting the resulting range allows mayBeTrueInRange to prune a file whose corrupt or differently encoded bounds do not cover its data, producing missing query results. Treat these suspicious bounds as unavailable and skip min/max pruning rather than taking a consequential fallback path.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

[ianton-ru]

This is not a P1. The overlapping band already fail-opens; converting |ticks| > 1e18 is the customer ns-stats case, not a guess inside the µs/ns overlap.
Iceberg timestamp is still microseconds in the schema. Magnitude cannot prove nanoseconds, which is why |ticks| in (1e16, 1e18] already returns nullopt. Callers then skip that hyperrectangle (if (!left || !right) continue) and min/max pruning for that column is skipped. That covers spec-correct far-future microseconds: Spark 9999-12-31 (~2.53e17 µs) and ClickHouse DateTime64 max 2299-12-31 (~1.04e16 µs).
Conversion runs only for |ticks| > 1e18. As microseconds that is ~year 33658, which is outside ClickHouse DateTime64(6) (max ~2299, ~1.04e16 µs) and outside normal Iceberg timestamp sentinels. As nanoseconds it is the customer layout (~1.76e18 for 2026) that previously over-pruned every file.
Fail-opening that band as well would keep query results correct (files scanned) but would disable min/max pruning again for those tables (IcebergMinMaxIndexPrunedFiles: 0). That undoes the point of this heuristic.
A corrupt 8-byte value > 1e18 that is not nanoseconds could still be converted into a fake in-range window. That is a residual, not a realistic P1 on this path.

[DimensionWieldr]

While writing some tests, AI found a potential issue:

Inserting into a format-version 3 table partitioned by DateTime64(9) fails with BAD_ARGUMENTS:

Unsupported type for iceberg DateTime64(9)

getAvroType in src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp still accepts only DateTime64 scale 6 when it encodes a partition value in the manifest. The data-file write path is fine: the same DateTime64(9) columns insert and prune on an unpartitioned v3 table.

This is the insert in test_writes_partition_pruning_datetime64_nanoseconds. Reproduced on 3948dca (clickhouse-common-static_26.6.4.20001.altinityantalya), table PARTITION BY ts with (ts DateTime64(9), value DateTime64(9), id Int32).

These cases on the same build succeed:

• Reading and pruning timestamp_ns, including identity partitions written outside ClickHouse.
• Unpartitioned DateTime64(9) writes stored as timestamp_ns, with min/max pruning.
• DateTime64(9, 'UTC') stored as timestamptz_ns.
• Create and ALTER of DateTime64(9) rejected on format versions 1 and 2; ALTER ADD Nullable(DateTime64(9)) allowed on version 3.

[ianton-ru]

@DimensionWieldr Thanks, must be fixed now

[DimensionWieldr]

CI looks good from my end.

Regression tests for timestamp nanosecond pruning are passing on both Ice and Glue catalogs: https://github.com/Altinity/clickhouse-regression/blob/main/iceberg/tests/iceberg_engine/timestamp_ns_pruning.py

[DimensionWieldr]

AI Audit Summary

Two follow-ups on timestamp_ns / timestamptz_ns.

---

DateTime64(9) with a non-UTC zone reloads as UTC. getIcebergType maps any DateTime64(9) with an explicit timezone to timestamptz_ns. IcebergSchemaProcessor::getSimpleType rebuilds that as DateTime64(9, "UTC"). shouldReloadSchemaForConsistency() returns true, and create plus every query rebuild columns from that mapping, so DateTime64(9, 'Europe/Berlin') becomes DateTime64(9, 'UTC') immediately. A string literal is parsed in the column zone, so WHERE ts = '2024-06-01 10:00:00' means 10:00 UTC afterward.

Iceberg does not keep the original zone name. Microsecond timestamptz already uses iceberg_timezone_for_timestamptz (default UTC). timestamptz_ns ignores that setting and hardcodes UTC. The read path should use the same setting.

---

timestamp_ns Parquet is marked UTC-adjusted. PrepareForWrite sets isAdjustedToUTC true for every DateTime64. Iceberg timestamp_ns requires TIMESTAMP(NANOS, false). An insert of DateTime64(9) with no zone writes timestamp_ns in the table metadata and a UTC-adjusted timestamp in the Parquet footer. DateTime64(9, 'UTC') is fine: that is timestamptz_ns, which is supposed to be adjusted.

ClickHouse reads the integers through the Iceberg schema, so a ClickHouse round-trip still succeeds. A reader that checks the Parquet logical type against the Iceberg type can reject the file. This flag was already forced true for every DateTime64, including ordinary timestamp. This change makes that mismatch apply to timestamp_ns. The fix belongs in the shared Parquet writer.

[DimensionWieldr]

Found one more thing:

Identity timestamp_ns partition fields lose the Avro logical type

Impact: Manifests for tables partitioned by DateTime64(9) store that partition key as a bare Avro long. CH still prunes using the table schema, but a reader that trusts the file schema does not see timestamp-nanos or adjust-to-utc, so it can treat nanosecond keys as an untyped integer.

Anchor: Utils.cpp / getAvroType; contrib/avro/lang/c++/impl/Compiler.cc / makeLogicalType; IcebergWrites.cpp / extendSchemaForPartitions

Trigger: INSERT into an Iceberg v3 table partitioned by DateTime64(9) or DateTime64(9, 'UTC').

Why defect: getAvroType emits logicalType: timestamp-nanos, then compileJsonSchemaFromString maps any unknown logical type to LogicalType::NONE. DataFileWriter persists that compiled schema, which omits the annotation. The comment in getAvroType says other engines still see it.

Fix direction (short): Keep the original schema JSON in the manifest (or teach the bundled Avro writer a timestamp-nanos logical type) so the written header still contains it.

[DimensionWieldr]

Debug build is failing.

The build is compiling unit tests, and two of them still call getIcebergType with two arguments. This PR changed that function so the third argument, format_version, is required.

[DimensionWieldr]

Also another possible issue:

Nanosecond bounds near the epoch are read as microseconds and can drop the whole file

Impact: A query can skip a data file that contains matching rows. If the file’s lower bound falls in a short window after the epoch and its upper bound is a normal nanosecond timestamp, the decoded range is inverted and every row in that file is pruned.

Anchor: src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp / deserializeDateTime64FromBinaryRepr

Trigger: Iceberg timestamp (DateTime64(6)) whose 8-byte min/max are nanoseconds, with the lower bound in about 1970-01-21..1970-04-26 and the upper bound after about 2001-09-09 (the same non-spec layout this PR fixes for 2024+ bounds).

Why defect: Only |ticks| > 1e18 is divided by 1000. |ticks| <= 1e16 is kept as microseconds (~115 days). A 2024 upper bound becomes ~1.7e15 µs while a lower bound of ~1e16 ns stays 1e16 µs, so lower > upper. Range::intersectsRange then matches nothing.

Fix direction (short): If either bound is in the nanosecond band, convert both, or fail open when the decoded lower bound is greater than the upper bound.

[ianton-ru]

Nanosecond bounds near the epoch are read as microseconds and can drop the whole file

Funny bug.

[DimensionWieldr]

More issues:

Manifest rewrite throws when a DateTime64(9) partition value is read back as Int64

Impact: OPTIMIZE manifest compaction of a table partitioned by DateTime64(9) / timestamp_ns fails with BAD_GET. The metadata commit has not run yet, so the table is left unchanged and the operation does not complete.

Anchor: getDateTime64ValueForIcebergBounds in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp, called from manifest-list partition summaries during compaction

Trigger: Format version 3 table, identity (or any transform whose result type is DateTime64(9)) partition, then manifest rewrite. contrib Avro has no timestamp-nanos logical type, so the manifest value comes back as Int64. Inserts still work because ChunkPartitioner produces a Decimal64.

Why defect: canDumpIcebergStats accepts the DateTime64 type, then safeGet() rejects the Int64 ticks. The manifest entry writer already accepts Int64; the summary path does not.

Fix direction (short): In getDateTime64ValueForIcebergBounds, treat Int64/UInt64 as already-scaled ticks, same as the decimal payload.

---

Nanosecond min/max stats within about 115 days of the epoch are decoded as microseconds and can drop the file

Impact: A query can skip a data file that contains matching rows when Iceberg timestamp (DateTime64(6)) lower/upper bounds were stored as nanoseconds. Results are missing, not just under-pruned.

Anchor: deserializeDateTime64FromBinaryRepr in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp, used by manifest min/max pruning

Trigger: Both bounds are nanosecond counts with absolute value <= 1e16 (about 1970-01-01 through 1970-04-26, and the matching pre-epoch window), and the predicate hits the real time. Example: one day after the epoch is 8.64e13 ns, which is kept as 8.64e13 µs (about year 1997). The inverted-range check does not fire when both ends stay in that band.

Why defect: The new decoder only divides by 1000 when |ticks| > 1e18. At or below 1e16 it treats the bytes as microseconds, so the hyperrectangle misses the real instants and mayBeTrueInRange prunes the file.

Fix direction (short): Fail open (skip this column’s min/max) for any DateTime64(6) bound that is not a proven microsecond value, or only convert when both bounds sit in the nanosecond band and still form a valid range.

[DimensionWieldr]

One more!

Nanosecond min/max on Iceberg timestamp still over-prunes a band around the epoch

Impact: A query can drop a data file and return fewer rows. Same failure this PR is fixing, for bounds whose integer magnitude also looks like a 2024–2286 microsecond timestamp.

Anchor: deserializeDateTime64FromBinaryRepr in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp

Trigger: Iceberg timestamp (DateTime64(6)) lower/upper bounds stored as nanoseconds with |ticks| in [1.704e15, 1e16]. That is about 1970-01-20 through 1970-04-26, and the negative mirror about 1969-09-07 through 1969-12-12. Both ends decode, the range stays ordered, and a predicate on the real instant does not overlap the decoded 2024–2286 range.

Why defect: Those ticks are kept as microseconds. 1e16 ns is ~115 days, and the same integer as microseconds is ~year 2286. Values below 2024-01-01 microseconds and the (1e16, 1e18] band fail open; this band does not.

src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergFieldParseHelpers.cpp lines 89-101

  if (scale == 6)
  {
      if (magnitudeBelow(unscaled, min_kept_microseconds))
          return std::nullopt;
      if (magnitudeExceeds(unscaled, microseconds_ambiguous_threshold))
      {
          if (!magnitudeExceeds(unscaled, nanoseconds_lower_threshold))
              return std::nullopt;
          unscaled = nanosecondsToMicrosecondsForBound(unscaled, lower_bound);
      }
  }

Fix direction (short): Fail open [min_kept_microseconds, 1e16] as well, or otherwise refuse to prune when a timestamp bound is in that overlap. No integer test separates those two scales.

[ianton-ru]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T13:29:05.151340Z	f884dae	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: f884dae729

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[DimensionWieldr]

CI is greeeeen 😭

AI audit (Grok 4.7) returned no more defects.

All of clickhouse-regression/iceberg/tests/iceberg_engine/timestamp_ns_pruning.py is passing.

LGTM

[ianton-ru]

Port on antalya-26.8 branch: #2610