Skip to content

fix(seed): respect declared column types when reading seed CSVs - #6136

Open
jackross wants to merge 2 commits into
SQLMesh:mainfrom
amfaro:fix/seed-csv-declared-types
Open

jackross wants to merge 2 commits into
SQLMesh:mainfrom
amfaro:fix/seed-csv-declared-types

Conversation

@jackross

Copy link
Copy Markdown

Description

Fixes #6135.

When a SEED model declares columns, its CSV was still parsed with pandas type inference, and the declared types were applied only afterwards in SeedModel.render_seed. That cast can't undo what inference already lost:

  • TEXT values like 00123 were loaded as '123', or '123.0' when the column also had an empty value.
  • Integer columns with an empty value were read as floats, losing precision above 2^53.

Fix

CsvSeedReader now receives the model's declared columns_to_types. It still reads the CSV exactly as before, then repairs only the declared columns that inference got wrong:

  • A declared TEXT column that pandas did not read as strings (it became numbers or booleans) is replaced by its raw strings.
  • A declared integer column that pandas read as floats is converted to exact Python ints and None. If any value isn't an integer literal (1.5, 1e3), the column keeps the floats pandas inferred, exactly as today.

The raw strings come from a second read_csv(dtype=str) with the same csv_settings. It runs only when at least one declared column needs repair.

I deliberately avoided passing dtype hints such as "Int64" to read_csv. In pandas 2.3, dtype="Int64" silently wraps values in [2^63, 2^64): 9223372036854775808 becomes -9223372036854775808. When the column also has a blank, the same values raise OverflowError. Reading as before and repairing afterwards means no CSV that loads today stops loading or changes, unless it hit this bug.

Behaviour changes

  • Declared TEXT columns keep values exactly as written in the CSV. Besides leading zeros, true stays true (previously True) and 1e3 stays 1e3 (previously 1000.0). Empty values and NA markers (na_values, keep_default_na) still load as NULL.
  • Declared integer columns that contain empty values load as integers and NULL instead of floats.
  • Booleans, floats/decimals, dates, integer columns without blanks, and undeclared columns are unchanged.
  • dbt seeds get the same handling, because their column types are passed as declared columns:
    • A column_types: {zip: varchar} override now keeps 00501. Previously it was loaded as '501', while dbt itself keeps '00501'.
    • On dbt ≥ 1.7, agate infers integer columns as int, so integer columns with blanks now load as integers instead of floats.
    • Agate-inferred decimals (double) are unaffected.

Upgrade note

Seed fingerprints include per-column hashes of the parsed data. Seeds that actually hit this bug get new column hashes, so they appear as modified once after upgrading and are reloaded with the corrected values. This covers declared TEXT columns pandas had read as numbers, and declared integer columns with blanks, including dbt seeds whose agate-inferred integer columns have blanks. Other seeds keep identical hashes; test_column_hashes_only_change_for_seeds_that_were_read_incorrectly covers both sides.

Declared TEXT columns with numeric-looking values and no leading zeros (e.g. 1, 2, 3) also change hash, even though their loaded values are the same. I couldn't find a cheap way to avoid this without keeping the old, incorrect parse around just for hashing.

The docs for seed models (docs/concepts/models/seed_models.md) gain a short paragraph explaining that declared columns control how CSV values are read.

Test Plan

New tests only; no existing assertions were changed.

  • tests/core/test_seed.py, reader level:
    • Leading zeros are kept, with and without blanks.
    • An integer column with a blank stays integer, and 9007199254740993 stays exact.
    • Non-integer literals keep today's floats.
    • Out-of-int64-range values, with and without a blank, read exactly as before.
    • csv_settings (delimiter, na_values) are honoured by the re-read.
    • Normalized-header matching works for Snowflake.
    • Column hashes change only for affected seeds.
  • tests/core/test_model.py::test_seed_model_declared_types_are_applied_when_reading_csv:
    • render_seed output for TEXT, INT, DOUBLE, BOOLEAN and DATE columns.
    • The exact rendered VALUES rows, e.g. ('00123', 1, 1.5, TRUE, CAST('2026-01-02' AS DATE)).
    • The rows stored in DuckDB.

I also reproduced the issue end to end in a DuckDB project before and after the change. Other engines receive the same DataFrame, and I did not test them against live warehouses.

Commands (Python 3.12, pandas 2.3.3):

  • make style: ruff, ruff-format, mypy and valid-migrations all pass.
  • make fast-test: 2675 passed, 6 skipped. Isolated groups: 5 passed, 1 passed and 190 passed.
  • pytest tests/dbt -k seed: 9 passed.
  • pytest tests/core/test_seed.py tests/core/test_model.py tests/core/test_snapshot.py: 503 passed.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Seed CSVs were parsed with pandas type inference and the declared
`columns` types were only applied afterwards, so information lost during
inference could not be recovered: TEXT values like 00123 were loaded as
'123' (or '123.0' when the column had an empty value), and integer
columns with empty values were read as floats, losing precision above
2**53.

When columns are declared, the CSV is still read as before, and declared
columns that inference got wrong are restored from a second read of the
raw strings: TEXT columns that weren't read as strings keep the values
as written, and integer columns read as floats are converted to exact
Python ints (left as is if a value isn't an integer literal). Columns
pandas already read correctly are untouched, so other seeds keep their
column hashes.

Signed-off-by: Jack A Ross <jack.ross@gmatter.co>
A TEXT column of TRUE/FALSE values that also has blanks is read by pandas
as an object column of Python bools, which the previous check mistook for
raw strings, so the values were rendered as 'True'/'False'. Restore a
declared TEXT column unless every value is already a string.

Signed-off-by: Jack A Ross <jack.ross@gmatter.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seed models parse CSVs before applying declared column types, so TEXT values lose leading zeros and integer columns become floats

1 participant