Skip to content

fix(snowflake): look up last modified timestamps in each table's database - #6132

Open
DG47 wants to merge 2 commits into
SQLMesh:mainfrom
DG47:fix/snowflake-last-modified-ts-cross-database
Open

DG47 wants to merge 2 commits into
SQLMesh:mainfrom
DG47:fix/snowflake-last-modified-ts-cross-database

Conversation

@DG47

@DG47 DG47 commented Oct 12, 2026

Copy link
Copy Markdown

Description

Fixes #6131.

SnowflakeEngineAdapter.get_table_last_modified_ts read LAST_ALTERED from the unqualified INFORMATION_SCHEMA.TABLES view. In Snowflake that view only lists tables in the session's current database, so tables in any other database were never returned. The freshness() signal then raised MissingSourceError for models that read external tables from a different database than the gateway's (for example RAW vs ANALYTICS).

This change groups the requested tables by database and queries each database's own INFORMATION_SCHEMA.TABLES ("RAW".INFORMATION_SCHEMA.TABLES, and so on). The TABLE_CATALOG predicate is no longer needed because each view is already scoped to one database. The BigQuery implementation groups per dataset in the same way. The query is now built with sqlglot rather than string formatting, so table and schema names are escaped as string literals. Tables without a catalog still go to the current database's information schema.

Test Plan

  • Added test_get_table_last_modified_ts_multiple_catalogs to tests/core/engine_adapter/test_snowflake.py. It requests three tables across two databases and checks that one query per database is issued and the timestamps are combined. It fails on main (one unqualified query) and passes with this change.
  • pytest tests/core/engine_adapter -m "not docker and not remote" -n 8: 714 passed, 3 skipped.
  • pytest -n 8 -m "fast and not web and not github and not dbt and not jupyter and not cicdonly and not isolated": 2436 passed, 6 failed. The 6 failures are the pyodbc connection-string tests, which fail locally on main too because unixODBC is not installed.
  • SKIP=prettier,eslint pre-commit run --all-files: ruff, ruff-format, mypy and valid migrations all pass.

The Snowflake freshness integration tests are skipped for Snowflake (_skip_snowflake in test_freshness.py), and I did not run this against a live Snowflake account.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

…base

SnowflakeEngineAdapter.get_table_last_modified_ts queried the unqualified
INFORMATION_SCHEMA.TABLES view, which only contains objects from the
connection's current database. Tables in any other database were never
found, so the freshness signal raised MissingSourceError for models that
read external tables from another database.

Group the tables by database and query each database's own
INFORMATION_SCHEMA.TABLES. The query is now built with sqlglot instead of
string formatting, so table and schema names are escaped as literals.

Signed-off-by: Dhruv Gupta <todhruvg@gmail.com>
@nicksspirit

nicksspirit commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Hey @DG47 — thanks for fixing this. Querying each database's own INFORMATION_SCHEMA is the right fix, and the new test covers the RAW vs ANALYTICS case from the issue.

One edge case: the tables are grouped by table.catalog, which drops quoting. In Snowflake, unquoted raw is database RAW, and quoted "raw" is a different database. If both appear in one call, they get grouped together and the query uses whichever catalog identifier came first.

Could you key the groups on the catalog identifier itself, so quoted and unquoted names stay separate? A small test that passes both "raw" and raw in one call would show they hit two different information schemas.

Thanks again for taking the time to put this up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snowflake: freshness signal cannot find external tables outside the connection's current database

2 participants