Repository navigation
[core] Compare FLOAT/DOUBLE partition values like Java - #1190
Open
JeonDaehong wants to merge 1 commit into
Open
JeonDaehong wants to merge 1 commit into
JeonDaehong wants to merge 1 commit into
Conversation
Row evaluation (partition filters, bucket filters) compared floats with partial_cmp, so every comparison with NaN was false and -0.0 equaled 0.0. Java Float.compare / Double.compare put every NaN above +Inf and -0.0 below 0.0. Manifest partition stats used the same <= / >=, so a NaN or -0.0 partition made min/max depend on entry order, and planning could drop partitions that match the filter.
Contributor
Author
|
@JingsongLi follow-up to #1174 for partition values. The manifest stats part can drop matching partitions depending on entry order, so I think it's worth a look. PTAL when you have time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #1189
Brief change log
eval_leafcompares FLOAT/DOUBLE in Java's order: every NaN is one value above+Inf, and-0.0is below0.0. Other types keepdatum_cmp.Datum's ownPartialEq/PartialOrdare unchanged.compute_partition_statsuses the same order, so the manifest's partition min/max no longer depend on entry order.The row-evaluation probe (8 values × 8 literals × 6 operators, FLOAT and DOUBLE, against paimon-bundle 2.1.0) went from 144 mismatches out of 768 to 0.
Tests
float_partition_filter_test: a DOUBLE-partitioned table with0.0,-0.0, NaN and1.0partitions, checking> 0.5,= NaN,!= NaNand= 0.0. On main,p > 0.5returns[3]or[]depending on the run.compute_partition_stats_orders_floats_like_java: four orderings of the same partitions give min-0.0and max NaN. With the old comparison, the first ordering gives min = max =1.0.test_non_roundtripping_candidates_keep_full_predicateexpected the NaN partition not to matchIN (NaN, 1.0). Java matches it, for a sign-bit NaN too.Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).
API and Format
None.
Documentation
None.