feat(sqlite): generate migrations (#51, fuzzed)

.changeset/quiet-sqlite-dialects.md

@@ -7,4 +7,4 @@ Add the `/sqlite` query entry with SQLite codecs, functions and positional bindi
 and explicit refusal of interactive transactions. `timestamp` stores SQLite's own
 `YYYY-MM-DD HH:MM:SS.SSS` UTC text, so it compares correctly with what `strftime` writes; `LIKE`
 escapes with `\` as on every dialect; text with U+0000 and integers past 2^53 are refused rather
-than silently changed. SQLite migration generation is not implemented yet.
+than silently changed.

.changeset/sqlite-generate.md

@@ -0,0 +1,10 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+`effect-orm generate` supports `dialect: "sqlite"`. It diffs `SQLite.table` definitions against
+the migrations folder and writes `<timestamp>_<name>/migration.sql` plus `snapshot.json`, the
+drizzle-kit 1.0 layout that Alchemy's D1 resource applies. `ALTER TABLE` adds, drops and renames
+where SQLite allows it; other column, primary key and UNIQUE changes rebuild the table. Renames
+need a `confirm_rename` hint. `generate --baseline --from-drizzle` adopts a drizzle-kit 1.0
+SQLite folder. Snapshots gain a `sqlite` dialect. `/migrate` refuses SQLite folders for now.

.changeset/sqlite-table-ddl.md

@@ -8,4 +8,4 @@ defaults (`SQLite.timestampNow` for a `timestamp` column), unique columns, and n
 unique indexes. `sqliteEntitiesOf` and
 `renderSqliteSchema` from `/schema` check a schema and write its `CREATE TABLE` and
 `CREATE INDEX` statements. `table(name, columns, options?)` and
-`table(name, { external: true, columns })` stay query-only. SQLite migrations are not generated yet.
+`table(name, { external: true, columns })` stay query-only.

README.md

@@ -9,7 +9,7 @@ Type-safe ClickHouse, Postgres and SQLite queries, result decoding, and reproduc
 Formerly `@maple-dev/effect-clickhouse`.
 
 SQLite and Cloudflare D1 use the same builder and row decoder through the new
-[`/sqlite` entry](./docs/sqlite.md), which also writes SQLite table DDL. SQLite migration generation is not implemented.
+[`/sqlite` entry](./docs/sqlite.md), which also writes SQLite table DDL. `effect-orm generate` writes SQLite migrations; effect-orm does not apply them yet.
 
 [Read the documentation](https://effect-clickhouse.maple.dev) ·
 [Getting started](./docs/getting-started.md) · [Recipes](./docs/recipes.md)

docs/README.md

@@ -102,4 +102,4 @@ See [Benchmarking](./benchmarking.md) and the
 
 ## SQLite and D1
 
-[SQLite and D1](./sqlite.md) covers `@maple-dev/effect-orm/sqlite`, its codecs and functions, table DDL, Effect SQL drivers, tenant checks and transaction limits. SQLite migration generation is not implemented.
+[SQLite and D1](./sqlite.md) covers `@maple-dev/effect-orm/sqlite`, its codecs and functions, table DDL, Effect SQL drivers, tenant checks and transaction limits. `effect-orm generate` writes SQLite migrations; effect-orm does not apply them yet.

docs/migrations.md

@@ -1,6 +1,6 @@
 # Schema and migrations
 
-Every table is declared with its DDL: `table` from `/clickhouse` or `/postgres` carries the
+Every table is declared with its DDL: `table` from `/clickhouse`, `/postgres` or `/sqlite` carries the
 engine, keys, indexes, and defaults beside the columns the query builder reads. You can ignore
 that and manage the database elsewhere, or let three opt-in entry points turn the definitions
 into migrations:
@@ -17,6 +17,7 @@ check and folder loading are shared; table definitions, the diff, the DDL, the r
 drift detection are per database, because the two differ where it matters: ClickHouse DDL is
 not transactional and cannot change most things in place, and Postgres DDL is and can. The
 sections below describe ClickHouse first; [Postgres](#postgres) covers what differs.
+[SQLite](#sqlite) has `generate` only: another tool applies its migrations for now.
 
 ## Defining tables
 
@@ -420,3 +421,95 @@ drizzle-kit recorded, so every place they disagree (a constraint name, a default
 op to accept or fix. `verify` against the baseline also finds objects the database has and
 drizzle-kit's snapshot does not, such as a table a hand-written migration created and nothing
 dropped.
+
+## SQLite
+
+Set `dialect: "sqlite"` and export `SQLite.table` values with their DDL from the schema module:
+
+```ts title="sqlite-kit-config.ts"
+import { defineConfig } from "@maple-dev/effect-orm/kit"
+
+export default defineConfig({ dialect: "sqlite", schema: "./src/db/schema.ts", out: "./migrations" })
+```
+
+`effect-orm generate` writes `<timestamp>_<name>/migration.sql` and `snapshot.json`, the
+drizzle-kit 1.0 layout. Statements are split by `--> statement-breakpoint`, each under a comment
+naming its label. A SQLite migration is always SQL; there is no `migration.json` for it. Tools
+that apply drizzle-kit folders apply it as it is, for example Alchemy's
+`Cloudflare.D1.Database({ migrations: "./migrations" })`, which runs each folder's statements as one
+D1 batch. `snapshot.json` is for `generate` and `check`; those tools ignore it.
+
+What each change becomes:
+
+| Change | SQL |
+| --- | --- |
+| New table, dropped table | `CREATE TABLE`, `DROP TABLE` (confirm the data loss) |
+| New column | `ALTER TABLE ... ADD COLUMN`, when SQLite allows it: the column is nullable or has a literal default, and is not in the primary key or a UNIQUE constraint. Otherwise a rebuild |
+| Dropped column | `ALTER TABLE ... DROP COLUMN` (confirm the data loss), unless it is in the primary key or a UNIQUE constraint. Otherwise a rebuild |
+| A column's type, nullability or default; the primary key; a UNIQUE column | A table rebuild. A new type also needs `confirm_data_loss` for the column: the copy converts values by SQLite's type affinity (`'007'` to `7`) |
+| New or changed index | `DROP INDEX`, `CREATE [UNIQUE] INDEX` |
+| Renamed table or column | `ALTER TABLE ... RENAME`, only with a `confirm_rename` hint |
+
+A rebuild follows [SQLite's procedure](https://www.sqlite.org/lang_altertable.html#otheralter)
+the way drizzle-kit writes it: `PRAGMA foreign_keys=OFF`, `CREATE TABLE "__new_<table>"` with the
+new definition, `INSERT INTO "__new_<table>" (...) SELECT ...` for every column both tables
+have, `DROP TABLE`, `ALTER TABLE "__new_<table>" RENAME TO "<table>"`, `PRAGMA foreign_keys=ON`,
+then every index of the table again. A column made NOT NULL that has a default is copied as
+`COALESCE(<column>, <default>)`, so existing NULLs take the default. A copy fails when the rows
+do not fit the new table, such as a column made NOT NULL over existing NULLs with no default, or
+a new UNIQUE constraint over duplicate values. A column a rebuild leaves out needs the same
+`confirm_data_loss` hint as a dropped column.
+
+> [!WARNING]
+> `PRAGMA foreign_keys=OFF` does nothing inside a transaction, and D1 and most migrators run each
+> migration in one. There, the rebuild's `DROP TABLE` deletes every row of the old table first, so
+> a table that references it with `ON DELETE CASCADE` loses its rows, and `ON DELETE SET NULL`
+> clears them. effect-orm declares no foreign keys, but a database may have them from another
+> tool. Before applying a rebuild of a referenced table, check for such keys
+> (`SELECT * FROM pragma_foreign_key_list('<child>')`) and, if there are any, run the rebuild
+> outside a transaction with foreign keys off.
+
+A rebuild renumbers rowids: the copy may read the old rows in any order. A table whose rows are
+referenced by rowid (rather than by a declared key) should declare an `integer` primary key, which
+is the rowid and is copied.
+
+`ALTER TABLE ... ADD COLUMN` always appends, so a column declared between two existing ones is
+last in the database and first-to-last order differs from a fresh create. Name columns in
+`INSERT` and `SELECT`, as the builder does.
+
+A rename is never guessed, except by case: SQLite compares table and column names without case,
+so `Url` to `url` (or `links` to `Links`) is the same name, renamed with no hint and no data lost.
+Any other rename without a hint reads as a drop plus an add, and the drop asks for confirmation.
+Pass the rename instead:
+
+```json
+[
+	{ "type": "confirm_rename", "kind": "table", "entity": "links", "to": "urls" },
+	{ "type": "confirm_rename", "kind": "column", "entity": "urls.url", "to": "target" }
+]
+```
+
+`entity` is the old table name, or `<table>.<old column>` with the table's new name. Table
+renames run first. Indexes keep their names and follow the rename.
+
+### Adopting a drizzle-kit SQLite folder
+
+`effect-orm generate --baseline --from-drizzle` reads drizzle-kit 1.0's last `snapshot.json`
+(version 7, `dialect: "sqlite"`): tables, columns, primary keys, indexes and UNIQUE
+constraints. Foreign keys, checks, views, generated and AUTOINCREMENT columns and partial
+indexes are listed, and nothing is written. The baseline runs nothing. A later `generate`
+compares your `SQLite.table` definitions with it; when they declare the same tables, it writes
+nothing. The tests prove this with a folder drizzle-kit 1.0.0-rc.5 generated.
+
+drizzle-kit 1.0 leaves `NOT NULL` out of a primary key column (`"id" text PRIMARY KEY`), so
+SQLite then accepts a NULL key in a `text` key column. Its snapshot marks the column
+`notNull: false`, but drizzle-kit treats a key column as not null. The baseline reads it as not
+null too, so a `SQLite.table` key column (always `NOT NULL`) matches it and no rebuild is
+written. The database keeps accepting a NULL key until that table is rebuilt for another change.
+
+The drizzle-kit 0.x layout (`meta/_journal.json` and flat `.sql` files) is not read. Run
+`drizzle-kit up` to convert it first.
+
+`/migrate` does not run SQLite folders yet: `run`, `status`, `verify` and `baseline` fail with a
+`MigrateSourceError` for one. Keep applying them with the tool you use now.
+

docs/reference.md

@@ -510,7 +510,7 @@ placeholders, double-quoted identifiers.
 | `SqlQuery`, `compileQuery`                                                                       | `/sql`            |
 | `ClickHouseStatement`, `parseStatement`, `renderStatement`, `withSettings`, `withFormat`         | `/sql`            |
 | `ClickHouseStatementFromString`, `splitTerminalClauses`, `maskLiteralsAndComments`               | `/sql`            |
-| Schema tooling: `isSchemaObject`, `makeSnapshot`, `renderSchema`, `diffSchemas`, `diffPgSchemas`, `fromDrizzleSnapshot`, snapshot and entity schemas (`PgEnumEntity`, `PgUniqueEntity`, …), `Hint`; for SQLite, `sqliteEntitiesOf`, `renderSqliteSchema`, `renderSqliteCreateTable`, `renderSqliteCreateIndex` and the `SqliteTableEntity`, `SqliteColumnEntity`, `SqliteIndexEntity`, `SqliteUniqueEntity` schemas | `/schema`         |
+| Schema tooling: `isSchemaObject`, `makeSnapshot`, `renderSchema`, `diffSchemas`, `diffPgSchemas`, `fromDrizzleSnapshot`, snapshot and entity schemas (`PgEnumEntity`, `PgUniqueEntity`, …), `Hint`; for SQLite, `sqliteEntitiesOf`, `renderSqliteSchema`, `renderSqliteCreateTable`, `renderSqliteCreateIndex`, `diffSqliteSchemas`, `fromDrizzleSqliteSnapshot`, `SqliteMigrationOp`, `renderSqliteOp`, `renderSqliteMigrationSql`, `labelOfSqlite`, `rebuildName`, the `SqliteSnapshot` schema and the `SqliteTableEntity`, `SqliteColumnEntity`, `SqliteIndexEntity`, `SqliteUniqueEntity` schemas | `/schema`         |
 | `run`, `status`, `verify`, `baseline`, `MigrationDriver` and the migrate errors                  | `/migrate`        |
 | `Database`, `run`, `sql`, `query`, `execute`, `transaction`, `requireTransaction`, `retryContention`, `Transaction` | `/database` |
 | `DatabaseError`, `TransactionCommitFailed`, `TransactionRollbackFailed` and the other transaction errors | `/database` |
@@ -594,5 +594,6 @@ drizzle-kit 1.0 writes and names it.
 **Types** — `ColumnInput`, `ColumnSpec`, `ColumnsOf`, `IndexSpec`, `TableDdl`, `SqliteSchemaTable`,
 `DdlExpr`, `DdlKey`, `DefinitionProblem`. A definition that cannot render records a problem;
 `S.sqliteEntitiesOf` fails with a `SchemaDefinitionError`, and `S.renderSqliteSchema` writes
-the `CREATE TABLE` and `CREATE INDEX` statements. There is no SQLite diff, snapshot or migration
-runner yet.
+the `CREATE TABLE` and `CREATE INDEX` statements. `effect-orm generate` with `dialect: "sqlite"`
+writes migrations for them; see [SQLite migrations](./migrations.md#sqlite). The `/migrate`
+runner does not apply SQLite migrations yet.

docs/sqlite.md

@@ -82,8 +82,11 @@ The tests apply this DDL and drizzle-kit's SQL for the same tables to real SQLit
 1.0 drops `NOT NULL` from a primary key column, so SQLite then accepts a NULL key in a `text`
 key column. This entry keeps `NOT NULL`.
 
-There is no SQLite schema diff, introspection, migration ledger or kit dialect yet. Run the
-rendered statements yourself, or keep migrating with your current tool. `WITHOUT ROWID`,
+`effect-orm generate` with `dialect: "sqlite"` diffs these tables against the migrations folder
+and writes the next migration, in the drizzle-kit 1.0 folder layout. See
+[SQLite migrations](./migrations.md#sqlite). effect-orm does not apply SQLite migrations yet,
+and has no SQLite introspection or drift check: apply the folder with a tool that reads
+drizzle-kit folders, such as Alchemy's D1 resource. `WITHOUT ROWID`,
 `STRICT`, `AUTOINCREMENT`, partial indexes, foreign keys and `CHECK` constraints are not
 declared yet.
 

src/kit/__fixtures__/drizzle-sqlite/20261025075810_init/migration.sql

@@ -0,0 +1,34 @@
+CREATE TABLE `membership` (
+	`org_id` text NOT NULL,
+	`user_id` text NOT NULL,
+	`role` text DEFAULT 'member' NOT NULL,
+	`weight` real DEFAULT 1.5 NOT NULL,
+	`active` integer DEFAULT true NOT NULL,
+	`avatar` blob,
+	`amount` numeric,
+	`email` text UNIQUE,
+	`created_at` text DEFAULT (CURRENT_TIMESTAMP) NOT NULL,
+	CONSTRAINT `membership_pk` PRIMARY KEY(`org_id`, `user_id`)
+);
+--> statement-breakpoint
+CREATE TABLE `short_link` (
+	`id` text PRIMARY KEY,
+	`code` text NOT NULL,
+	`target_url` text NOT NULL,
+	`target_hash` text NOT NULL,
+	`org_id` text,
+	`created_by` text,
+	`subject_type` text NOT NULL,
+	`subject_id` text NOT NULL,
+	`purpose` text NOT NULL,
+	`prospect_id` text,
+	`click_count` integer NOT NULL,
+	`first_accessed_at` text,
+	`last_accessed_at` text,
+	`created_at` text NOT NULL,
+	`updated_at` text NOT NULL
+);
+--> statement-breakpoint
+CREATE INDEX `membership_user_role` ON `membership` (`user_id`,`role`);--> statement-breakpoint
+CREATE UNIQUE INDEX `short_link_code` ON `short_link` (`code`);--> statement-breakpoint
+CREATE INDEX `short_link_org` ON `short_link` (`org_id`);