Skip to content

Inconsistencies between RecordBatch and DataFrame schemas cause to_arrow_table to fail #1314

Closed
#1315
Closed
Inconsistencies between RecordBatch and DataFrame schemas cause to_arrow_table to fail#1314
#1315
Labels
bugSomething isn't working

Description

@nuno-faria
nuno-faria
opened on Dec 1, 2025

Describe the bug

When the nullability of a RecordBatch column does not match with the DataFrame's schema, the conversion to a pyarrow table fails.

To Reproduce

from datafusion import SessionContext

ctx = SessionContext()
ctx.sql("create table t_(a int not null)").collect()
ctx.sql("insert into t_ values (1), (2), (3)").collect()
ctx.sql(f"copy (select * from t_) to 't.parquet'").collect()
ctx.register_parquet("t", path)
pyarrow_table = ctx.sql("select max(a) as m from t").to_arrow_table()
...
pyarrow.lib.ArrowInvalid: Schema at index 0 was different: 
m: int32
vs
m: int32 not null

Expected behavior
Execute without crashing.

Additional context

Activity

  1. nuno-faria
    added
    bugSomething isn't working
    on Dec 1, 2025

  2. IshaGudewar commented on Dec 4, 2025

    @IshaGudewar
    IshaGudewar
    on Dec 4, 2025

    Thanks for opening this issue and the fix!
    I reviewed the problem and the proposed solution in PR #1315, and I can confirm that the root cause is the schema mismatch between:

    the DataFrame schema (which may mark fields as NOT NULL), and
    the RecordBatch schema (which often marks aggregated columns as nullable).

    PyArrow requires all RecordBatch schemas to match exactly, so this mismatch causes:

    ArrowInvalid: Schema at index 0 was different

    To help strengthen the PR, I will add:

    1. A regression test for the zero-record-batch case, verifying that when no batches are returned, to_arrow_table() still uses the DataFrame schema.

    2. A test for an aggregation that returns NULL, such as max(a) on an empty table, confirming the output is [None] and schema nullability is handled correctly.

    I will contribute these tests to PR #1315 shortly.

  3. nuno-faria commented on Dec 4, 2025

    @nuno-faria
    nuno-faria
    on Dec 4, 2025
    ContributorAuthor

    Thanks @IshaGudewar, but those tests have already been added.

  4. timsaucer
    closed this as completedin #1315on Jan 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions