[GeorgiaPreda]

Hey! I have an RFC for #9184.

Summary

Pandas 3 retains object dtype when concatenating an all-null object column with a typed column. P2P shuffle instead promotes Arrow's null type to the typed column, losing that distinction and replacing original null sentinels.

Implementation and tradeoff

• Add temporary uint8 null-sentinel columns for null-bearing object and relevant extension columns, with schema metadata identifying the columns and whether object promotion is required.
• Merge that metadata across Arrow concatenations and remove all temporary columns before returning the Pandas result.
• Preserve large integers during conversion instead of passing them through floating point.
• Remove the existing expected-failure marker and cover both disk and memory shuffles, input ordering, distinct missing values, nullable integers, large integer precision, column ordering, and helper-name collisions.

Validation

python -m pytest distributed/shuffle/tests/test_shuffle.py distributed/shuffle/tests/test_merge.py -k "(null_partitions or null_object_column or processing_chain or split_by_worker or split_by_partition or basic_integration or handle_object_columns or stable_ordering or basic_merge or merge_by_multiple_columns) and not local_cluster"

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±  0      40 suites  ±0   14h 9m 58s ⏱️ + 17m 31s
 4 203 tests + 43   4 021 ✅ + 40    178 💤 ±  0  4 ❌ +3 
81 734 runs  +775  77 303 ✅ +584  4 427 💤 +188  4 ❌ +3 

For more details on these failures, see this check.

Results for commit 21c444f. ± Comparison against base commit dc182bd.

This pull request removes 1 and adds 44 tests. Note that renamed tests count towards both.

distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2

distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2[False-False]
distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2[False-True]
distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2[True-False]
distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2[True-True]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values0-False]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values0-True]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values1-False]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values1-True]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values2-False]
distributed.shuffle.tests.test_shuffle ‑ test_null_object_column_survives_arrow_concatenation[Int64-null_values2-True]
…