[JeonDaehong]

Purpose

Linked issue: close #1174

Brief change log

The residual filter compares FLOAT/DOUBLE with Arrow kernels, which order floats by their bits. A NaN with the sign bit set, which is what x86 produces for 0.0 / 0.0, sorts below -Inf there and is not equal to NaN. Java's Float.compare / Double.compare treat every NaN as one value above +Inf.

evaluate_column_predicate now rewrites such NaNs in the column to NaN before comparing. The FLOAT/DOUBLE literal is canonicalized the same way. A column without a non-canonical NaN is used as-is.

A probe of 8 values × 8 literals × 6 operators for FLOAT and DOUBLE against paimon-bundle 2.1.0 PredicateBuilder went from 112 mismatches out of 768 to 0.

The BTree global index orders keys by bits too, so with only the residual change a table with a BTree index would answer differently. NaN literals are now canonicalized, and a BTree file whose keys hold NaNs leaves a predicate to the row filter when its key order can disagree with Java: ranges when the first key is a NaN, NaN literals when either end is. Files without NaN keys keep exact answers.

Row evaluation (eval_row) is not changed here (see #1174).

Tests

float_negative_nan_filter_test writes one file with 0.0, a sign-bit NaN, -2.0 and NaN, and checks <, >, BETWEEN, = NaN, != NaN, = -NaN and IN on FLOAT and DOUBLE. Without the change, f < -1.0 returns [1, 2].

test_bitmap_floating_candidates_preserve_residual_results (#601) used the residual's old answers as its baseline, for example a sign-bit NaN matching < 0 and = NaN matching only that exact bit pattern. Its expected rows now follow Java. The bitmap index still returns the same rows as the residual in every case.

test_bitmap_floating_candidates_preserve_residual_results now also runs on BTree and covers > / >=. test_evaluate_legacy_{float,double}_btree and test_bitmap_floating_bound_fusion_preserves_candidates_and_fallback expected bit-order answers for NaN literals and bounds, and now expect the Java result.

Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None.

Documentation

None.

[JeonDaehong]

@JingsongLi follow-up to your note in #1160, PTAL when you have time.

[JeonDaehong]

Added a second commit after checking BTree. BTree keys keep total_cmp order, so once the residual follows Java, a table with a BTree index answered differently from a scan without one. > 0 and >= -1 dropped sign-bit NaN rows, and = NaN / IN (NaN) matched only one NaN payload. On main the two agree because both use bit order.

The fix keeps exact BTree answers wherever the key order can't matter. NaN literals are canonicalized. Only a file whose keys hold NaNs, at its first key for ranges or at either end for NaN literals, leaves that predicate to the row filter. Meta pruning and the all-match shortcut skip such files too.

test_bitmap_floating_candidates_preserve_residual_results now also runs on BTree and covers > / >=. test_evaluate_legacy_{float,double}_btree and test_bitmap_floating_bound_fusion_preserves_candidates_and_fallback pinned bit-order answers for NaN literals and bounds, so they now expect the Java result.