Skip to content

Preserve field metadata through physical CASE expressions - #125

Merged
osipovartem merged 7 commits into
embucket-sync-df55.0.0from
df55-physical-case-metadata
Oct 9, 2026
Merged

osipovartem merged 7 commits into
embucket-sync-df55.0.0from
df55-physical-case-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Port apache/datafusion#26158 to our pinned DF55 fork. Physical CASE preserves agreed Arrow field metadata, including nested CASE and casts. The planner retains the logical result field when physical lowering loses metadata context, and physical-expression protobuf round-trips that field. The common unmarked Column/Literal path is unchanged.

The matching logical CASE change was merged into embucket-sync-df55.0.0 via #126. This branch is rebased onto that base. The Rustice integration uses a separately pinned branch containing both changes.

Validation

  • cargo +stable test -p datafusion-physical-expr --features proto --lib --quiet: 1,688 passed, 2 ignored on this branch.
  • Regression tests cover matching/conflicting metadata, NULL and typed-NULL branches, nested CASE and casts, alias-lowered marked UDFs, scalar variables, marked NOT, child rewrites, and protobuf encode/decode.
  • The combined metadata.slt SQLLogicTest passed on the rebased branch, including direct and projected CASE metadata checks.
  • Formatting, diff checks, and focused Clippy passed. Independent read-only review approved the rebased branch at 54db65b1c with no blockers.
  • Nested CASE planning at depths 16/32/64/128 measured approximately 0.4/1.3/4.5/17.1 ms in the shared implementation. There is no per-row execution overhead; the common unmarked fast path remains unchanged.

Upstream: apache#26158

@osipovartem
osipovartem force-pushed the df55-physical-case-metadata branch from 82a027c to 54db65b Compare October 9, 2026 15:06
@osipovartem
osipovartem marked this pull request as ready for review October 9, 2026 15:07
@osipovartem
osipovartem merged commit 5e2f790 into embucket-sync-df55.0.0 Oct 9, 2026
36 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant