Skip to content

fix: preserve field metadata for simple CASE branches - #26145

Draft
osipovartem wants to merge 19 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata
Draft

osipovartem wants to merge 19 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Part of #24860. This is logical schema propagation for CASE, complementary to #24865.

What changes?

  • Preserve Arrow field metadata when supported CASE result branches agree on type and metadata. Untyped NULL does not erase it; typed NULL carrying matching metadata remains eligible.
  • Handle columns, literals, casts, aliases, and nested CASE iteratively. The ordinary unmarked CASE fast path is unchanged.
  • Preserve metadata returned by shallow scalar/higher-order functions (including a shallow nested CASE argument) and explicit-metadata Cast, TryCast, and Alias wrappers using their existing to_field contract. Exact delegation is limited to expression depth 8 to bound repeated field inference. Deeper function branches and unsupported branch forms conservatively leave CASE metadata empty.
  • Physical CaseExpr metadata and constant-false simplification remain separate follow-ups to CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860.

Validation

  • Local datafusion-expr and datafusion-optimizer suites: 283 + 934 tests passed. Focused CASE tests, formatting, and datafusion-expr test-target Clippy also pass.
  • SQL integration test passes in both Apache and Embucket builds. It checks planned and optimized logical field metadata for searched and simple CASE, matching, NULL, and conflicting result branches, plus executes a result-row smoke test. The latest focused core_integration run passed and cargo fmt --all --check passed.
  • Focused regression covers matching/conflicting metadata, absent ELSE, multiple WHEN branches, typed/untyped NULL, marked and unmarked literals, negative and unsupported-NOT expressions, nested CASE/cast, shallow nested CASE within a metadata-bearing UDF, explicit Cast/TryCast/Alias wrappers, deep fallback, outer references, scalar variables, placeholders, lambda variables, first-branch NULL, and type mismatch.
  • Independent read-only reviews approved correctness, bounded planning cost, API compatibility, and expression-level and SQL-level test scope.
  • The last posted Codecov comment measured 74ac5de5b at 74.06% patch coverage. Additional branch-focused tests were added in cc221b5fd, d03385576, and 3a8631b7f; a report for the current head is still pending. Some old partials are defensive errors and Rust ? branches in tests. No coverage percentage is claimed for the current head yet.

@codecov-commenter

codecov-commenter commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.40395% with 137 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.76%. Comparing base (4e6e564) to head (7cc7ac7).

Files with missing lines Patch % Lines
datafusion/expr/src/expr_schema.rs 73.93% 39 Missing and 89 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 86.36% 2 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26145      +/-   ##
==========================================
- Coverage   82.77%   82.76%   -0.02%     
==========================================
  Files        1148     1148              
  Lines      451075   451632     +557     
  Branches   451075   451632     +557     
==========================================
+ Hits       373397   373810     +413     
- Misses      54930    54976      +46     
- Partials    22748    22846      +98     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osipovartem
osipovartem marked this pull request as draft October 8, 2026 23:00
@osipovartem osipovartem changed the title fix: preserve common field metadata through CASE expressions fix: preserve field metadata for simple CASE branches Oct 8, 2026
@osipovartem
osipovartem marked this pull request as ready for review October 9, 2026 08:58
(cherry picked from commit 816119e839317cca1b31ef02ad0e8e302e0c60d0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants