Skip to content

feat(sqlite): generate migrations - #51

Closed
artile wants to merge 25 commits into
MapleTechLabs:mainfrom
artile:feat/sqlite-generate
Closed

artile wants to merge 25 commits into
MapleTechLabs:mainfrom
artile:feat/sqlite-generate

Conversation

@artile

@artile artile commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #50 (stacked; the diff includes #49 and #50 until they merge). Refs #48.

Changes

  • Snapshot and SchemaDialect gain sqlite (SqliteSnapshot). AnySchemaEntity includes the SQLite entities, so the branch graph, check and snapshot hashing work for SQLite folders unchanged.
  • effect-orm generate with dialect: "sqlite" diffs SQLite.table definitions against the folder and writes <timestamp>_<name>/migration.sql and snapshot.json. The SQL is split by --> statement-breakpoint, each statement under a -- <label> comment. SQLite never gets a migration.json.
  • New diffSqliteSchemas and SqliteMigrationOp (/schema):
    • CREATE TABLE, DROP TABLE (needs confirm_data_loss).
    • ALTER TABLE ADD COLUMN when SQLite allows it: nullable or a literal default, not in the key or a UNIQUE constraint.
    • ALTER TABLE DROP COLUMN (needs confirm_data_loss) unless the column is in the key or a UNIQUE constraint.
    • Every other change (type, nullability, default, primary key, UNIQUE, a column ADD COLUMN cannot add) is one rebuild_table op: PRAGMA foreign_keys=OFF, CREATE TABLE "__new_<t>", INSERT ... SELECT of the shared columns, DROP TABLE, ALTER TABLE ... RENAME TO, PRAGMA foreign_keys=ON, then the table's indexes again. This is drizzle-kit's form of SQLite's documented procedure.
    • DROP INDEX / CREATE [UNIQUE] INDEX.
    • ALTER TABLE ... RENAME TO and RENAME COLUMN only from a confirm_rename hint. Hint accepts kind: "table" | "column" next to "enum_value". A hint that does not fit both schemas is reported as unsupported.
  • generate --baseline --from-drizzle reads a drizzle-kit 1.0 SQLite snapshot.json (version 7) with the new fromDrizzleSqliteSnapshot: tables, columns, keys, indexes, UNIQUE constraints. Anything else (foreign keys, checks, views, generated or AUTOINCREMENT columns, partial indexes) is listed and nothing is written. dialectOf now reads the dialect a drizzle-kit snapshot names, instead of assuming Postgres.
  • /migrate run, status, verify, baseline and resolveStep fail with a MigrateSourceError for a SQLite folder. Without this they would use the ClickHouse ledger.
  • Docs: a SQLite section in docs/migrations.md, updates to docs/sqlite.md, docs/reference.md, the README lines and the CLI help, and a minor changeset.

Validation

  • bun install --frozen-lockfile, bun run build, bun run typecheck: ok.
  • bun run test with a local ClickHouse 26.8.2.7 in Docker: 65 files, 1075 tests passed, 0 skipped. Doc citations ok (26 files), export catalog ok (287 /clickhouse, 205 /postgres, 167 /sqlite), doc examples ok (26 files).
  • bun scripts/check-package.ts: ESM and CommonJS consumers ok; the ESM consumer diffs and snapshots a SQLite schema from the tarball.
  • src/kit/sqlite-generate.test.ts (16 tests) runs generate in a temporary folder. For each change it applies the migrations in order to :memory: and compares PRAGMA table_info, index_list and index_xinfo with a fresh create of the final schema. Seeded rows must survive. Changes covered: first create, add column, drop column, type, nullability, default, UNIQUE, expression-default column, primary key, index add/drop, table drop, a column a rebuild leaves out, table and column rename.
  • The same file applies a generated rebuild migration to a local D1 database (Miniflare), one batch per migration, and checks rows and indexes.
  • Adoption: src/kit/__fixtures__/drizzle-sqlite is a folder drizzle-kit 1.0.0-rc.5 generated for short_link and a second table (composite key, defaults, a UNIQUE column). Its source schema is next to it. Baseline, then generate from the equal SQLite.table definitions: no change. One changed index then produces exactly its drop and create.

Scope and limits

  • No SQLite runner, ledger or verify (U4); /migrate refuses SQLite folders on purpose.
  • The drizzle-kit 0.x layout (meta/_journal.json) is not read; Alchemy refuses it too and points to drizzle-kit up.
  • Renames are never inferred. Without a hint, a rename reads as a drop plus an add, and the drop asks for confirmation.
  • A rebuild copies only the shared columns. It fails at run time when rows do not fit, for example NOT NULL over existing NULLs or a new NOT NULL column without a default. This matches how a Postgres migration fails.
  • PRAGMA foreign_keys=OFF/ON is a no-op inside a transaction. effect-orm models no foreign keys yet, so the pragmas only matter for keys created elsewhere.

Decisions to review

  • drizzle-kit 1.0 writes a text primary key without NOT NULL and records notNull: false, so SQLite accepts a NULL key. The importer reads a key column as not null, which is what drizzle-kit means and what SQLite.table writes. A drizzle-created table then matches with no rebuild, but still accepts a NULL key until it is rebuilt for another change. A future verify must treat this case as known.
  • confirm_rename for a column takes <new table name>.<old column>, so it composes with a table rename in the same migration.
  • The baseline migration.sql is a comment only, as for Postgres. Alchemy sends it in one batch with its own bookkeeping statement.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

artile added 25 commits October 24, 2026 14:49
@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c837e1ce-fc56-4cb9-ad0c-048b8ca2edee
Commits

Reviewing files that changed from the base of the PR and between c5fa679 and b6e2067.

Files ignored due to path filters: 1
  • bun.lock is excluded by !**/*.lock
Files selected for processing: 49
  • .changeset/quiet-sqlite-dialects.md
  • .changeset/sqlite-generate.md
  • .changeset/sqlite-table-ddl.md
  • README.md
  • docs/README.md
  • docs/migrations.md
  • docs/reference.md
  • docs/sqlite.md
  • package.json
  • scripts/check-exports-documented.mjs
  • src/ch/compile.ts
  • src/ch/dialect.ts
  • src/ch/expr.ts
  • src/ch/functions/builtin.ts
  • src/kit/__fixtures__/drizzle-sqlite/20261025075810_init/migration.sql
  • src/kit/__fixtures__/drizzle-sqlite/20261025075810_init/snapshot.json
  • src/kit/__fixtures__/drizzle-sqlite/schema.ts.txt
  • src/kit/cli.ts
  • src/kit/generate.ts
  • src/kit/sqlite-generate.test.ts
  • src/migrate/run.ts
  • src/migrate/source.ts
  • src/migrate/verify.ts
  • src/schema.ts
  • src/schema/ddl-expr.ts
  • src/schema/diff.ts
  • src/schema/entities.ts
  • src/schema/pg-define.ts
  • src/schema/snapshot.ts
  • src/schema/sqlite-define.ts
  • src/schema/sqlite-diff.ts
  • src/schema/sqlite-drizzle.test.ts
  • src/schema/sqlite-drizzle.ts
  • src/schema/sqlite-entities.ts
  • src/schema/sqlite-ops.ts
  • src/schema/sqlite-render.ts
  • src/sql/sql-syntax.ts
  • src/sqlite.ts
  • src/sqlite/d1.test.ts
  • src/sqlite/ddl.test.ts
  • src/sqlite/dialect.ts
  • src/sqlite/functions.ts
  • src/sqlite/shape.test-support.ts
  • src/sqlite/sqlite.test-d.ts
  • src/sqlite/sqlite.test.ts
  • src/sqlite/types.ts
  • tests/package-consumer-declarations.mts
  • tests/package-consumer.mts
  • tsdown.config.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Makisuo

Makisuo commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Thanks! #121 has your commits unchanged plus a migration fuzzer (random schema pairs, run on seeded rows, checked against a fresh create). It found two bugs. First, a rebuild that keeps no column ({c2} → {c0}) wrote no INSERT and deleted every row. Second, replacing every column ({c1,c2} → {c3}) dropped the last column before adding, which SQLite refuses. Both are fixed in #121, along with COALESCE(col, default) for a column made NOT NULL with a default, and a docs warning about ON DELETE CASCADE firing during a rebuild under D1. It's stacked on #120 → #119. Pull the commits here if you prefer, or we'll land the stack.

Makisuo added a commit that referenced this pull request Oct 25, 2026
* feat(sqlite): generate writes a first SQLite migration and snapshot

* feat(sqlite): add columns with ALTER TABLE ADD COLUMN

* feat(sqlite): rebuild tables for changes ALTER TABLE cannot make

* feat(sqlite): rename tables and columns from confirm_rename hints

* feat(sqlite): adopt a drizzle-kit 1.0 folder with generate --baseline --from-drizzle

* test(sqlite): apply generated migrations to a local D1 database

* fix(migrate): refuse SQLite folders until the runner has a SQLite ledger

* docs(kit): name SQLite in the CLI help

* docs(sqlite): document generated SQLite migrations and drizzle-kit adoption

* docs(sqlite): explain the SQLite diff in its header

* fix(sqlite): migrations keep rows, never drop a table's last column

Found by fuzzing migrations between random schemas on seeded rows:

- A rebuild that keeps no column (every old column dropped, new ones
  added) wrote no INSERT, so it deleted every row though only the
  columns' data was confirmed lost. It now copies the rowids, and each
  row takes the new columns' defaults.
- Replacing every column of a table dropped them before adding the new
  ones, and SQLite cannot drop a table's last column. Column adds now
  run before column drops.
- A column made NOT NULL that has a default failed the rebuild over
  existing NULLs: INSERT ... SELECT passes an explicit NULL, which does
  not take the default. The copy is COALESCE(column, default) now.

docs/migrations.md: a rebuild under D1 or any transaction fires ON DELETE
CASCADE in tables that reference the rebuilt one (PRAGMA foreign_keys=OFF
does nothing there), a rebuild renumbers rowids, and ADD COLUMN appends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuzz): migrations between random SQLite schemas reach the fresh schema and keep rows

Two random schemas over a shared pool of table and column names: new,
dropped and changed tables, columns, types, nullability, literal and
expression defaults, keys, UNIQUE columns and indexes. The migration
diffSqliteSchemas writes runs inside a transaction on rows that fit
both schemas. The PRAGMA shape must equal a fresh create of the new
schema (column order aside) and every row must survive, holding its
values or the new column's default.

Pins the three counterexamples fixed in the previous commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuzz): migrations on D1 and with renames; drizzle-kit adoption round trip

- The migration property runs on D1 too: each migration as one batch on a
  local workerd binding, as Alchemy's D1 resource applies it, compared
  with a fresh create on a second D1 database.
- Inputs rename a shared table and one of its kept columns through
  confirm_rename hints (about half of all inputs do), so renames combine
  with every other change and with rebuilds.
- A drizzle-kit 1.0 snapshot of any random schema (written as drizzle-kit
  writes it, key columns `notNull: false` included) imports with nothing
  left to generate.
- Changesets no longer say SQLite DDL or migrations are not implemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sqlite): confirm type changes in a rebuild; rename by case without a hint

From an independent review:
- A rebuild that changes a kept column's type converts its values by
  SQLite's type affinity ('007' to 7, '1e3' to 1000) with no prompt. It
  now needs confirm_data_loss for that column, like a drop.
- A column or table renamed by case only (Url to url) read as a drop and
  an add, which then collided: SQLite names ignore case. The diff now
  renames it, with no hint, since nothing is lost. SQLite refuses a
  case-only table rename in one step, so it goes through a temporary name.
- The migration fuzzer renames by case without hints too, and confirms
  drops and type changes under old and new names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuzz): the D1 migration engine retries transport errors only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Taras Kornichuk <dread9ko@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Makisuo

Makisuo commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Merged as part of #121 (8eb7200 on main), with your commits kept as they were and review fixes on top. Thank you, @tao-io, this is a great contribution! The SQLite migration generator now ships with the fuzzer and shared core suite running on SQLite and D1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants