Repository navigation
feat(sqlite): table DDL — columns, keys, indexes #50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
1969a67
feat: add SQLite and D1 codecs and dialect profiles
artile ceb1f05
feat: preserve SQLite offset and union branch pagination
artile 4fc495d
feat: reject SQL clauses SQLite cannot execute
artile f74a9db
fix: disambiguate SQLite insert-select upserts
artile 80cf8a9
feat: add schema-aware SQLite aggregate and scalar functions
artile 65e4bfa
test: verify SQLite transactions and local D1 safety checks
artile 20b9478
docs: publish the SQLite entry and document its tested limits
artile 0e96f31
docs: list SQLite under ON CONFLICT in the reference
artile bfe02f8
fix(sqlite): build external tables from their columns
artile 23cb739
feat(sqlite): table DDL with columns, primary key and indexes
artile e82620d
feat(sqlite): column defaults, unique columns and composite primary keys
artile 7951133
test(sqlite): DDL definition problems, expression keys and query-only…
artile 16e23f3
docs(sqlite): document table DDL and check it from the packed tarball
artile d5e6a52
docs: mention SQLite table DDL in the README and docs index
artile 1a5aa0e
feat(sqlite): write unique columns as UNIQUE constraints, as drizzle-…
artile File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| "@maple-dev/effect-orm": minor | ||
| --- | ||
|
|
||
| Add the `/sqlite` query entry with SQLite codecs, functions and positional bindings. Use | ||
| `sqliteDialect` with an Effect SQLite SqlClient, or `d1Dialect` for Cloudflare D1's binding limit | ||
| and explicit refusal of interactive transactions. `table` is a query schema only: SQLite DDL and | ||
| migration generation are not implemented yet. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| --- | ||
| "@maple-dev/effect-orm": minor | ||
| --- | ||
|
|
||
| `SQLite.table(name, { columns, primaryKey, indexes })` declares a SQLite table with its DDL: | ||
| `NOT NULL` from the column type, single and composite primary keys, literal and expression | ||
| defaults, unique columns, and named indexes and unique indexes. `sqliteEntitiesOf` and | ||
| `renderSqliteSchema` from `/schema` check a schema and write its `CREATE TABLE` and | ||
| `CREATE INDEX` statements. `table(name, columns, options?)` and | ||
| `table(name, { external: true, columns })` stay query-only. SQLite migrations are not generated yet. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| # SQLite and Cloudflare D1 | ||
|
|
||
| Import `@maple-dev/effect-orm/sqlite` for the shared query builder, SQLite codecs and functions. | ||
| Compilation uses positional `?` bindings. Each occurrence of a parameter gets its own binding. | ||
| The package opens no connection: use the Effect `SqlClient` you already provide. | ||
|
|
||
| ## Define and compile a query | ||
|
|
||
| ```ts title="sqlite-quickstart.ts" | ||
| import * as SQLite from "@maple-dev/effect-orm/sqlite" | ||
|
|
||
| export const Acks = SQLite.table("acknowledgments", { | ||
| id: SQLite.text, | ||
| org_id: SQLite.text, | ||
| entry_id: SQLite.text, | ||
| }, { tenantColumn: "org_id" }) | ||
|
|
||
| export const byOrg = SQLite.from(Acks) | ||
| .select("entry_id") | ||
| .where(($) => [$.org_id.eq(SQLite.param.string("orgId"))]) | ||
| .orderBy(["entry_id", "asc"]) | ||
|
|
||
| export const compiled = SQLite.compileUnsafe(byOrg, { orgId: "org_1" }) | ||
| // compiled.parameters is ["org_1"]; compiled.tenantScope is "single-tenant". | ||
| ``` | ||
|
|
||
| `table(name, columns, options?)` is a query schema only: it carries no DDL. | ||
|
|
||
| ## Declare a table with its DDL | ||
|
|
||
| Pass `{ columns, primaryKey, indexes }` to `table` to declare the table's DDL too. The value is | ||
| still a query `Table`. `sqliteEntitiesOf` from `/schema` checks a set of tables as one schema, | ||
| and `renderSqliteSchema` writes their `CREATE TABLE` and `CREATE INDEX` statements: | ||
|
|
||
| ```ts title="sqlite-ddl.ts" | ||
| import { Effect } from "effect" | ||
| import * as S from "@maple-dev/effect-orm/schema" | ||
| import * as SQLite from "@maple-dev/effect-orm/sqlite" | ||
|
|
||
| export const ShortLinks = SQLite.table("short_link", { | ||
| columns: { | ||
| id: SQLite.text, | ||
| code: SQLite.text, | ||
| orgId: SQLite.column(SQLite.nullable(SQLite.text), { name: "org_id" }), | ||
| clickCount: SQLite.column(SQLite.integer, { name: "click_count", default: 0 }), | ||
| createdAt: SQLite.column(SQLite.text, { name: "created_at", defaultExpr: "(CURRENT_TIMESTAMP)" }), | ||
| }, | ||
| primaryKey: ["id"], | ||
| indexes: [SQLite.uniqueIndex("short_link_code", ["code"]), SQLite.index("short_link_org", ["orgId"])], | ||
| tenantColumn: "orgId", | ||
| }) | ||
|
|
||
| export const statements = S.sqliteEntitiesOf([ShortLinks]).pipe(Effect.map(S.renderSqliteSchema)) | ||
| // CREATE TABLE IF NOT EXISTS "short_link" ( | ||
| // "id" text PRIMARY KEY NOT NULL, | ||
| // "code" text NOT NULL, | ||
| // "org_id" text, | ||
| // "click_count" integer NOT NULL DEFAULT 0, | ||
| // "created_at" text NOT NULL DEFAULT (CURRENT_TIMESTAMP) | ||
| // ) | ||
| // CREATE UNIQUE INDEX IF NOT EXISTS "short_link_code" ON "short_link" ("code") | ||
| // CREATE INDEX IF NOT EXISTS "short_link_org" ON "short_link" ("org_id") | ||
| ``` | ||
|
|
||
| - A column is `NOT NULL` unless its type is `nullable(...)`. A primary key column cannot be | ||
| nullable: SQLite would accept NULL keys in it. | ||
| - One primary key column is written inline (`PRIMARY KEY`); several become a table constraint | ||
| `PRIMARY KEY ("a", "b")`. SQLite stores no constraint name, so there is none to give. | ||
| - `default` takes a literal of the column type. `defaultExpr` takes SQL as written, so add the | ||
| parentheses SQLite requires around an expression, or a DSL callback, which is parenthesized | ||
| for you. Either lets an insert leave the column out. | ||
| - `unique: true` writes an inline `UNIQUE` constraint, as drizzle-kit 1.0 does for `.unique()`. | ||
| Its name, `<table>_<column>_unique` or the string you give, is kept in the schema entities; | ||
| SQLite itself names the backing index `sqlite_autoindex_...`. | ||
| - Column types declare their SQLite type: `text`, `timestamp` and `json` are `text`; `integer` | ||
| and `boolean` are `integer`; `real` is `real`; `blob` is `blob`. A `custom(sql, schema)` type | ||
| declares `sql`, for example `numeric`. | ||
| - `table(name, { external: true, columns })` declares a table another tool owns: no DDL. | ||
|
|
||
| The tests apply this DDL and drizzle-kit's SQL for the same tables to real SQLite, and compare | ||
| `PRAGMA table_info`, `index_list` and `index_xinfo`. One difference is deliberate: drizzle-kit | ||
| 1.0 drops `NOT NULL` from a primary key column, so SQLite then accepts a NULL key in a `text` | ||
| key column. This entry keeps `NOT NULL`. | ||
|
|
||
| There is no SQLite schema diff, introspection, migration ledger or kit dialect yet. Run the | ||
| rendered statements yourself, or keep migrating with your current tool. `WITHOUT ROWID`, | ||
| `STRICT`, `AUTOINCREMENT`, partial indexes, foreign keys and `CHECK` constraints are not | ||
| declared yet. | ||
|
|
||
| The entry exports `text`, `integer` (a safe JavaScript integer), `real` (finite numbers), | ||
| `boolean` (0/1), `blob`, `timestamp` (ISO text), `json(schema)`, `nullable`, `custom` and `brand`. | ||
| `blob` accepts the Uint8Array wire form from Node and the array-of-bytes form from D1. | ||
| `json` always uses the supplied schema to validate decoded JSON. | ||
| Use `text` when a stored date/time must remain a string. | ||
|
|
||
| SQLite aggregates include `count`, `countDistinct`, `countIf`, `sum`, `avg`, `min` and `max`. | ||
| The last four return NULL over an empty input. Scalar helpers include `lower`, `upper`, | ||
| `length` and `coalesce`. SQLite's own text and case rules apply. Foreign built-in function | ||
| sets, such as ClickHouse's `count()`, fail compilation instead of producing invalid SQL. | ||
|
|
||
| ## Execute on SQLite | ||
|
|
||
| Install a compatible Effect SQL driver, for example `@effect/sql-sqlite-node` on Node 24+. | ||
| No driver is a runtime dependency of this package. Use current SQLite with RETURNING support. | ||
|
|
||
| ```ts title="sqlite-node.ts" | ||
| import * as SqliteClient from "@effect/sql-sqlite-node/SqliteClient" | ||
| import { Layer } from "effect" | ||
| import * as Db from "@maple-dev/effect-orm/database" | ||
| import { sqliteDialect } from "@maple-dev/effect-orm/sqlite" | ||
|
|
||
| export const SQLiteLive = Db.layerSqlClient({ | ||
| dialect: sqliteDialect, | ||
| requireTenantScope: true, | ||
| }).pipe(Layer.provide(SqliteClient.layer({ filename: ":memory:" }))) | ||
| ``` | ||
|
|
||
| `Db.run(query, params)` compiles for the database dialect and decodes the selected rows. | ||
| SQLite transactions and nested savepoints use the supplied client's `withTransaction`. | ||
| `serializable` is the only accepted explicit isolation level. Per-transaction access modes | ||
| and deferrable settings are refused. The driver must support the declared transaction behavior; | ||
| use the D1 profile for a D1 client. | ||
|
|
||
| ## Execute on D1 | ||
|
|
||
| Install `@effect/sql-d1` with a compatible Effect version. Provide the Worker's D1 binding: | ||
|
|
||
| ```ts title="sqlite-d-one.ts" | ||
| import * as D1Client from "@effect/sql-d1/D1Client" | ||
| import { Layer } from "effect" | ||
| import * as Db from "@maple-dev/effect-orm/database" | ||
| import { d1Dialect } from "@maple-dev/effect-orm/sqlite" | ||
|
|
||
| export const d1DatabaseLayer = (db: D1Client.D1ClientConfig["db"]) => | ||
| Db.layerSqlClient({ dialect: d1Dialect, requireTenantScope: true }).pipe( | ||
| Layer.provide(D1Client.layer({ db })), | ||
| ) | ||
| ``` | ||
|
|
||
| Use `d1Dialect` when compiling a query separately as well: | ||
| `SQLite.compile(query, params, { dialect: SQLite.d1Dialect })`. A query compiled for `sqlite` | ||
| cannot later run on a database configured as `d1`; pass the builder to `Db.run` instead. | ||
|
|
||
| D1 compilation refuses more than [100 bound parameters per statement](https://developers.cloudflare.com/d1/platform/limits/). | ||
| It never splits a write implicitly. Chunk an import explicitly if needed; several chunks do | ||
| not make one atomic write. SQLite compilation uses its modern default limit of 32766; a driver | ||
| with a lower limit needs a custom dialect profile. | ||
|
|
||
| `Database.transaction` on D1 returns `TransactionUnsupported` before executing its body. | ||
| D1's [atomic batch API](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch) | ||
| executes a fixed list of statements. It cannot run an arbitrary Effect transaction body with | ||
| reads that determine later writes. There is no ORM batch API in this first implementation. | ||
|
|
||
| ## Safety and syntax limits | ||
|
|
||
| Derived schemas validate returned rows. A row with the wrong type returns | ||
| `CompiledQueryDecodeError`. Input values use each column's encoder before binding. | ||
|
|
||
| Tenant enforcement is **opt-in**, through `requireTenantScope: true`. A compiled query over a | ||
| tenant table must pin its `tenantColumn` to one value, or explicitly use `.crossTenant()` for | ||
| an administrative read/write. Raw SQL passed to `query` or `execute` has no tenant proof and | ||
| bypasses this guard. It is not a replacement for an application's authorization checks. | ||
|
|
||
| The dialect handles OFFSET without LIMIT, branch-local ordering/limits in UNION ALL, and | ||
| INSERT SELECT with ON CONFLICT. It refuses these unsupported forms at compile time: | ||
|
|
||
| - `DISTINCT ON`, row locks and `ILIKE`. Use `LIKE` with SQLite's own case rules. | ||
| - `ON CONSTRAINT` conflict targets. Name the indexed columns instead. | ||
| - Per-cell `DEFAULT` in a VALUES tuple. Omit a defaulted column from every row in the statement, | ||
| or group rows by their present columns. All-default rows are not supported by the builder. | ||
| - ClickHouse FORMAT/SETTINGS and built-in functions from another dialect. | ||
|
|
||
| The tests run real SQLite through `@effect/sql-sqlite-node` and local workerd/D1 through | ||
| Miniflare. They require no deployed database or Cloudflare credentials. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Release note contradicts the second changeset.
Line 7 and Line 8 say
tableis a query schema only and that SQLite DDL is not implemented..changeset/sqlite-table-ddl.mdin this stack addsSQLite.table(name, { columns, primaryKey, indexes })with DDL. If both changesets ship in one release, the changelog states both that DDL exists and that it does not.Drop the DDL claim from this note. Keep the statement that migration generation is not implemented.
📝 Proposed fix
Committable suggestion