Skip to content

[core] Compare FLOAT/DOUBLE partition values like Java - #1190

Open
JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:fix-row-eval-float-order
Open

JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:fix-row-eval-float-order

Conversation

@JeonDaehong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #1189

Brief change log

  • eval_leaf compares FLOAT/DOUBLE in Java's order: every NaN is one value above +Inf, and -0.0 is below 0.0. Other types keep datum_cmp. Datum's own PartialEq / PartialOrd are unchanged.
  • compute_partition_stats uses the same order, so the manifest's partition min/max no longer depend on entry order.

The row-evaluation probe (8 values × 8 literals × 6 operators, FLOAT and DOUBLE, against paimon-bundle 2.1.0) went from 144 mismatches out of 768 to 0.

Tests

  • float_partition_filter_test: a DOUBLE-partitioned table with 0.0, -0.0, NaN and 1.0 partitions, checking > 0.5, = NaN, != NaN and = 0.0. On main, p > 0.5 returns [3] or [] depending on the run.
  • compute_partition_stats_orders_floats_like_java: four orderings of the same partitions give min -0.0 and max NaN. With the old comparison, the first ordering gives min = max = 1.0.
  • test_non_roundtripping_candidates_keep_full_predicate expected the NaN partition not to match IN (NaN, 1.0). Java matches it, for a sign-bit NaN too.

Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None.

Documentation

None.

Row evaluation (partition filters, bucket filters) compared floats with
partial_cmp, so every comparison with NaN was false and -0.0 equaled 0.0.
Java Float.compare / Double.compare put every NaN above +Inf and -0.0 below
0.0. Manifest partition stats used the same <= / >=, so a NaN or -0.0
partition made min/max depend on entry order, and planning could drop
partitions that match the filter.
@JeonDaehong

Copy link
Copy Markdown
Contributor Author

@JingsongLi follow-up to #1174 for partition values. The manifest stats part can drop matching partitions depending on entry order, so I think it's worth a look. PTAL when you have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FLOAT/DOUBLE partition filters and manifest partition stats disagree with Java on NaN and -0.0

1 participant