Repository navigation
Support equality with one in DF55 WindowTopN safely - #127
osipovartem wants to merge 1 commit into
Conversation
|
Likely implementation-level source of the measured high-cardinality regression (not yet a CPU profile): DF55 The right follow-up is to benchmark and port the shared-store approach (or an equivalent bounded-memory design) with focused high-cardinality correctness, memory, and spill tests. Avoid enabling this draft rule in Rustice before that evidence exists. The preceding five-run 123/644 ms off/on result is the acceptance blocker. |
|
Closing this draft. Rustice already registers its own |
What changes
rank_col = 1and1 = rank_colin the opt-in DF55 WindowTopN rule; leave other equalities unchanged.LEAD) depends on pruned rows.ORDER BYon the partition key itself. This also closes a pre-existing panic path for inequality predicates.Upstream counterpart: apache#26160. The DF55 branch needs the sibling and effective-order guards that already exist in current upstream.
Validation
window_topn.sltSQLLogicTest passed.cargo +1.95.0 test --profile ci -p datafusion-physical-optimizer --lib: 34 passed.cargo +1.95.0 clippy --profile ci -p datafusion-physical-optimizer --all-targets -- -D warningspassed.cargo +1.95.0 fmt --all --checkpassed.Performance blocker
Five local 100k-row CLI runs, median query elapsed (opt-in rule off/on):
The high-cardinality case is about 5.2x slower. This PR remains draft and must not be merged or pinned into Rustice until that regression has an acceptable mitigation and benchmark evidence. The default
enable_window_topnflag remains false. A pushed-down FilterExec projection can also block the rewrite; this change does not claim a Snowplow speedup.