Skip to content

Preserve list element metadata when rebuilding scalar values - #26092

Open
osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:fix/list-scalar-field-metadata
Open

osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:fix/list-scalar-field-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ScalarValue::cast_to reconstructs LIST values through SingleRowListArrayBuilder::with_field. That builder copied the child field name and nullability but dropped its metadata. Casting a LIST to a target field containing nested metadata (for example an Iceberg PARQUET:field_id) therefore produced a scalar with a different type from the requested target, and subsequent batch expansion could fail schema validation.

Copy the field metadata as well. This only runs when constructing a scalar LIST; columnar cast and batch fast paths are unchanged.

Validation

  • New regression test covers LIST scalar cast to a metadata-bearing element field and batch expansion.
  • cargo +1.95.0 test --offline -p datafusion-common --lib: 624 passed.
  • cargo +1.95.0 fmt --all --check: passed.
  • Independent read-only review: approved, no blocking findings.

@github-actions github-actions Bot added the common Related to common crate label Oct 6, 2026
@codecov-commenter

codecov-commenter commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.77%. Comparing base (d137137) to head (d4e2791).

Files with missing lines Patch % Lines
datafusion/common/src/utils/mod.rs 89.47% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26092      +/-   ##
==========================================
- Coverage   82.78%   82.77%   -0.01%     
==========================================
  Files        1148     1148              
  Lines      451075   451090      +15     
  Branches   451075   451090      +15     
==========================================
+ Hits       373400   373401       +1     
- Misses      54927    54935       +8     
- Partials    22748    22754       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants