feat: add SQLite and Cloudflare D1 query dialects

.changeset/quiet-sqlite-dialects.md

@@ -0,0 +1,8 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+Add the `/sqlite` query entry with SQLite codecs, functions and positional bindings. Use
+`sqliteDialect` with an Effect SQLite SqlClient, or `d1Dialect` for Cloudflare D1's binding limit
+and explicit refusal of interactive transactions. `table` is a query schema only: SQLite DDL and
+migration generation are not implemented yet.

README.md

@@ -4,10 +4,13 @@
 > **Alpha.** The API is still changing and minor releases may break it. Pin an exact version,
 > read the [changelog](./CHANGELOG.md) before upgrading, and expect rough edges.
 
-Type-safe ClickHouse and Postgres queries, result decoding, and reproducible benchmarks for Effect and TypeScript.
+Type-safe ClickHouse, Postgres and SQLite queries, result decoding, and reproducible benchmarks for Effect and TypeScript.
 
 Formerly `@maple-dev/effect-clickhouse`.
 
+SQLite and Cloudflare D1 use the same builder and row decoder through the new
+[`/sqlite` entry](./docs/sqlite.md). SQLite schema/migration generation is not implemented.
+
 [Read the documentation](https://effect-clickhouse.maple.dev) ·
 [Getting started](./docs/getting-started.md) · [Recipes](./docs/recipes.md)
 

docs/README.md

@@ -99,3 +99,7 @@ The optional `@maple-dev/effect-orm/benchmark` entry point and bundled
 `ch-bench` CLI measure real queries, compare fixed workloads, and save evidence.
 See [Benchmarking](./benchmarking.md) and the
 [agent playbook](./benchmark-agent.md). The SQL builders remain driver-free.
+
+## SQLite and D1
+
+[SQLite and D1](./sqlite.md) covers `@maple-dev/effect-orm/sqlite`, its codecs and functions, Effect SQL drivers, tenant checks and transaction limits. SQLite migration generation is not implemented.

docs/reference.md

@@ -5,9 +5,10 @@ The package has one entry per database, each self-contained:
 ```ts
 import * as CH from "@maple-dev/effect-orm/clickhouse"
 import * as PG from "@maple-dev/effect-orm/postgres"
+import * as SQLite from "@maple-dev/effect-orm/sqlite"
 ```
 
-Both carry the whole [query builder](#query-builder-both-entries). Each adds its own column
+All carry the whole [query builder](#query-builder-every-entry). Each adds its own column
 types, functions, table definitions and a `compile` that defaults to its dialect:
 [`/clickhouse`](#clickhouse) and [`/postgres`](#postgres). Everything else is on a
 [subpath](#other-subpaths) for a narrower job.
@@ -32,9 +33,9 @@ uniformly, which some codebases prefer for exactly this reason.
 
 ---
 
-## Query builder (both entries)
+## Query builder (every entry)
 
-Everything in this section is exported, identically, from both `/clickhouse` and `/postgres`.
+Everything in this section is exported, identically, from `/clickhouse`, `/postgres` and `/sqlite`.
 
 ### Query construction
 
@@ -46,7 +47,7 @@ Everything in this section is exported, identically, from both `/clickhouse` and
 | `unionAll`  | `(...queries) => CHUnionQuery`            |
 | `update`     | `(table) => CHUpdateStart`, then `CHUpdate`: `.set(record \| fn)`, `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)`. See [Updating and deleting](./updates-and-deletes.md) |
 | `deleteFrom` | `(table) => CHDelete`: `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)` |
-| `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) |
+| `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres and SQLite). See [Inserting rows](./inserts.md) |
 
 Tables come from the dialect's `table`: [`CH.table`](#tables-and-ddl) or
 [`PG.table`](#tables-and-ddl-1).
@@ -530,3 +531,26 @@ verification, and JSON protocol.
 Note `/sql` exports a `compile` (fragment → string) distinct from the dialect entries' `compile`
 (query → `CompiledQuery`), and a `when` distinct from the query builder's `when` (optional
 conditions).
+
+## SQLite and D1
+
+`@maple-dev/effect-orm/sqlite` exports the shared query builder and the four compile functions.
+`sqliteDialect` is the default; `d1Dialect` uses the same SQL with a 100-binding limit and no
+interactive transactions. See [SQLite and D1](./sqlite.md) for driver wiring and limits.
+
+| Export | Purpose |
+| --- | --- |
+| `SqliteType` | A SQLite SQL type with its Effect codec |
+| `text`, `integer`, `real` | Text, safe integer and finite number codecs |
+| `boolean` | A boolean encoded as 0/1; other integers fail decoding |
+| `timestamp` | An instant stored as ISO text |
+| `blob` | Bytes from a Uint8Array or D1's array wire representation |
+| `json(schema)` | JSON text decoded through the supplied schema |
+| `nullable(type)`, `custom`, `brand` | Type composition |
+| `table(name, columns, options?)`, `TableOptions` | A query schema, including `tenantColumn`, defaults and computed-column metadata; no generated DDL |
+| `table(name, ExternalTableDefinition)`, `ExternalTableDefinition` | `{ external: true, columns, tenantColumn? }`, the same query schema in the `/clickhouse` and `/postgres` form |
+| `count`, `countDistinct`, `countIf`, `sum`, `avg`, `min`, `max` | SQLite aggregates; the last four can return SQL NULL |
+| `lower`, `upper`, `length`, `coalesce` | SQLite scalar functions |
+
+SQLite's table definition is for query typing and decoding. The schema/kit/migrate entries
+currently support ClickHouse and Postgres DDL only.

docs/sqlite.md

@@ -0,0 +1,115 @@
+# SQLite and Cloudflare D1
+
+Import `@maple-dev/effect-orm/sqlite` for the shared query builder, SQLite codecs and functions.
+Compilation uses positional `?` bindings. Each occurrence of a parameter gets its own binding.
+The package opens no connection: use the Effect `SqlClient` you already provide.
+
+## Define and compile a query
+
+```ts title="sqlite-quickstart.ts"
+import * as SQLite from "@maple-dev/effect-orm/sqlite"
+
+export const Acks = SQLite.table("acknowledgments", {
+	id: SQLite.text,
+	org_id: SQLite.text,
+	entry_id: SQLite.text,
+}, { tenantColumn: "org_id" })
+
+export const byOrg = SQLite.from(Acks)
+	.select("entry_id")
+	.where(($) => [$.org_id.eq(SQLite.param.string("orgId"))])
+	.orderBy(["entry_id", "asc"])
+
+export const compiled = SQLite.compileUnsafe(byOrg, { orgId: "org_1" })
+// compiled.parameters is ["org_1"]; compiled.tenantScope is "single-tenant".
+```
+
+`table` describes the query schema. Manage its actual DDL and migrations outside this package.
+Unlike the ClickHouse and Postgres entries, SQLite has no schema diff, introspection, migration
+ledger or kit dialect yet. A `table` value here cannot be passed to the DDL generators.
+
+The entry exports `text`, `integer` (a safe JavaScript integer), `real` (finite numbers),
+`boolean` (0/1), `blob`, `timestamp` (ISO text), `json(schema)`, `nullable`, `custom` and `brand`.
+`blob` accepts the Uint8Array wire form from Node and the array-of-bytes form from D1.
+`json` always uses the supplied schema to validate decoded JSON.
+Use `text` when a stored date/time must remain a string.
+
+SQLite aggregates include `count`, `countDistinct`, `countIf`, `sum`, `avg`, `min` and `max`.
+The last four return NULL over an empty input. Scalar helpers include `lower`, `upper`,
+`length` and `coalesce`. SQLite's own text and case rules apply. Foreign built-in function
+sets, such as ClickHouse's `count()`, fail compilation instead of producing invalid SQL.
+
+## Execute on SQLite
+
+Install a compatible Effect SQL driver, for example `@effect/sql-sqlite-node` on Node 24+.
+No driver is a runtime dependency of this package. Use current SQLite with RETURNING support.
+
+```ts title="sqlite-node.ts"
+import * as SqliteClient from "@effect/sql-sqlite-node/SqliteClient"
+import { Layer } from "effect"
+import * as Db from "@maple-dev/effect-orm/database"
+import { sqliteDialect } from "@maple-dev/effect-orm/sqlite"
+
+export const SQLiteLive = Db.layerSqlClient({
+	dialect: sqliteDialect,
+	requireTenantScope: true,
+}).pipe(Layer.provide(SqliteClient.layer({ filename: ":memory:" })))
+```
+
+`Db.run(query, params)` compiles for the database dialect and decodes the selected rows.
+SQLite transactions and nested savepoints use the supplied client's `withTransaction`.
+`serializable` is the only accepted explicit isolation level. Per-transaction access modes
+and deferrable settings are refused. The driver must support the declared transaction behavior;
+use the D1 profile for a D1 client.
+
+## Execute on D1
+
+Install `@effect/sql-d1` with a compatible Effect version. Provide the Worker's D1 binding:
+
+```ts title="sqlite-d-one.ts"
+import * as D1Client from "@effect/sql-d1/D1Client"
+import { Layer } from "effect"
+import * as Db from "@maple-dev/effect-orm/database"
+import { d1Dialect } from "@maple-dev/effect-orm/sqlite"
+
+export const d1DatabaseLayer = (db: D1Client.D1ClientConfig["db"]) =>
+	Db.layerSqlClient({ dialect: d1Dialect, requireTenantScope: true }).pipe(
+		Layer.provide(D1Client.layer({ db })),
+	)
+```
+
+Use `d1Dialect` when compiling a query separately as well:
+`SQLite.compile(query, params, { dialect: SQLite.d1Dialect })`. A query compiled for `sqlite`
+cannot later run on a database configured as `d1`; pass the builder to `Db.run` instead.
+
+D1 compilation refuses more than [100 bound parameters per statement](https://developers.cloudflare.com/d1/platform/limits/).
+It never splits a write implicitly. Chunk an import explicitly if needed; several chunks do
+not make one atomic write. SQLite compilation uses its modern default limit of 32766; a driver
+with a lower limit needs a custom dialect profile.
+
+`Database.transaction` on D1 returns `TransactionUnsupported` before executing its body.
+D1's [atomic batch API](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch)
+executes a fixed list of statements. It cannot run an arbitrary Effect transaction body with
+reads that determine later writes. There is no ORM batch API in this first implementation.
+
+## Safety and syntax limits
+
+Derived schemas validate returned rows. A row with the wrong type returns
+`CompiledQueryDecodeError`. Input values use each column's encoder before binding.
+
+Tenant enforcement is **opt-in**, through `requireTenantScope: true`. A compiled query over a
+tenant table must pin its `tenantColumn` to one value, or explicitly use `.crossTenant()` for
+an administrative read/write. Raw SQL passed to `query` or `execute` has no tenant proof and
+bypasses this guard. It is not a replacement for an application's authorization checks.
+
+The dialect handles OFFSET without LIMIT, branch-local ordering/limits in UNION ALL, and
+INSERT SELECT with ON CONFLICT. It refuses these unsupported forms at compile time:
+
+- `DISTINCT ON`, row locks and `ILIKE`. Use `LIKE` with SQLite's own case rules.
+- `ON CONSTRAINT` conflict targets. Name the indexed columns instead.
+- Per-cell `DEFAULT` in a VALUES tuple. Omit a defaulted column from every row in the statement,
+  or group rows by their present columns. All-default rows are not supported by the builder.
+- ClickHouse FORMAT/SETTINGS and built-in functions from another dialect.
+
+The tests run real SQLite through `@effect/sql-sqlite-node` and local workerd/D1 through
+Miniflare. They require no deployed database or Cloudflare credentials.

package.json

@@ -1,15 +1,17 @@
 {
 	"name": "@maple-dev/effect-orm",
 	"version": "0.7.2",
-	"description": "Type-safe ClickHouse and Postgres queries, result decoding, and reproducible benchmarks for Effect and TypeScript",
+	"description": "Type-safe ClickHouse, Postgres and SQLite queries, result decoding, and reproducible benchmarks for Effect and TypeScript",
 	"keywords": [
 		"clickhouse",
 		"effect",
 		"orm",
 		"postgres",
 		"query-builder",
 		"sql",
-		"typescript"
+		"typescript",
+		"sqlite",
+		"cloudflare-d1"
 	],
 	"homepage": "https://effect-clickhouse.maple.dev",
 	"bugs": {
@@ -97,6 +99,11 @@
 			"types": "./dist/benchmark/cli.d.mts",
 			"import": "./dist/benchmark/cli.mjs",
 			"default": "./dist/benchmark/cli.mjs"
+		},
+		"./sqlite": {
+			"types": "./dist/sqlite.d.mts",
+			"import": "./dist/sqlite.mjs",
+			"default": "./dist/sqlite.mjs"
 		}
 	},
 	"publishConfig": {
@@ -121,13 +128,16 @@
 		"@effect/language-service": "^0.87.3",
 		"@effect/platform-bun": "4.0.0",
 		"@effect/sql-clickhouse": "4.0.0",
+		"@effect/sql-d1": "4.0.0",
 		"@effect/sql-pglite": "4.0.0",
+		"@effect/sql-sqlite-node": "4.0.0",
 		"@effect/vitest": "4.0.0",
 		"@electric-sql/pglite": "0.5.8",
 		"@types/node": "^22.10.2",
 		"effect": "4.0.0",
 		"expect-type": "^1.3.0",
 		"fast-check": "^4.9.0",
+		"miniflare": "4.20260730.0",
 		"tsdown": "^0.22.14",
 		"typescript": "^7.0.2",
 		"vitest": "^4.1.9"

scripts/check-exports-documented.mjs

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-// Every export of the two dialect entries is named in docs/reference.md, whose
+// Every export of the three dialect entries is named in docs/reference.md, whose
 // first line claims to be the catalog. A claim like that decays the moment
 // anything is added, so it is checked rather than promised.
 //
@@ -11,7 +11,7 @@ import { dirname, join } from "node:path"
 import { fileURLToPath } from "node:url"
 
 const root = join(dirname(fileURLToPath(import.meta.url)), "..")
-const entries = ["clickhouse", "postgres"]
+const entries = ["clickhouse", "postgres", "sqlite"]
 
 const exportsOf = (entry) => {
 	const dts = join(root, `dist/${entry}.d.mts`)

src/ch/compile.ts

@@ -936,7 +936,9 @@ function compileInner<
 			const sqlQuery: SqlQuery = {
 				distinct: state.distinct !== undefined,
 				distinctOn: Array.isArray(state.distinct)
-					? (state.distinct.length === 0
+					? (currentDialect().clauses.distinctOn === false
+						? fail<ReadonlyArray<string>>(new QueryBuilderDefect({ message: `${currentDialect().name} has no DISTINCT ON clause` }), [])
+						: state.distinct.length === 0
 							? fail<ReadonlyArray<string>>(new QueryBuilderDefect({ message: "CHQuery: distinctOn() needs at least one key" }), [])
 							: state.distinct
 						).map((key: string) => {
@@ -967,7 +969,9 @@ function compileInner<
 				orderBy: state.orderByFn === undefined
 					? orderByClause(state.orderBySpecs).map(raw)
 					: orderByExprs(state.orderByFn($)).map(raw),
-				limit: rowCount("limit", state.limitValue),
+				limit: state.limitValue == null && state.offsetValue != null && currentDialect().clauses.offsetRequiresLimit === true
+					? raw("-1")
+					: rowCount("limit", state.limitValue),
 				offset: rowCount("offset", state.offsetValue),
 				format: options?.skipFormat ? undefined : formatClause(state.formatValue),
 			}
@@ -1442,7 +1446,9 @@ function compileUnionInner<Output extends Record<string, any>, Params extends Re
 				: "cross-tenant"
 
 	const parenthesize = currentDialect().clauses.parenthesizeUnionBranches
-	let sql = subQueries.map((q) => (parenthesize ? `(\n${q.sql}\n)` : q.sql)).join("\nUNION ALL\n")
+	let sql = subQueries.map((q) => currentDialect().clauses.wrapUnionBranches === true
+		? `SELECT * FROM (\n${q.sql}\n) AS ${quoteIdent("__union_branch")}`
+		: parenthesize ? `(\n${q.sql}\n)` : q.sql).join("\nUNION ALL\n")
 
 	// Wrap in outer SELECT if ordering/pagination is needed
 	const hasOuter =
@@ -1455,6 +1461,8 @@ function compileUnionInner<Output extends Record<string, any>, Params extends Re
 		}
 		if (state.outerLimitValue != null) {
 			sql += `\nLIMIT ${compileSqlFragment(rowCount("limit", state.outerLimitValue)!)}`
+		} else if (state.outerOffsetValue != null && currentDialect().clauses.offsetRequiresLimit === true) {
+			sql += "\nLIMIT -1"
 		}
 		if (state.outerOffsetValue != null) {
 			sql += `\nOFFSET ${compileSqlFragment(rowCount("offset", state.outerOffsetValue)!)}`
@@ -1684,6 +1692,9 @@ const onConflictClause = (
 	let target = ""
 	if (conflict.target !== undefined) {
 		if ("constraint" in conflict.target) {
+			if (dialect.clauses.onConflictConstraint === false) {
+				return fail(new QueryBuilderDefect({ message: `${where}: ${dialect.name} has no ON CONSTRAINT conflict target; name its columns instead` }), "")
+			}
 			if (conflict.targetWhere !== undefined) {
 				return fail(new QueryBuilderDefect({ message: `${where}: targetWhere needs a column target, not a constraint` }), "")
 			}
@@ -1778,7 +1789,9 @@ const valueCells = (table: Table<string, ColumnDefs>, params: Record<string, unk
 	const values: Record<string, unknown> = { ...params }
 	let next = 0
 	const cell = (column: string, value: unknown, context: string): string => {
-		if (value === undefined) return "DEFAULT"
+		if (value === undefined) return currentDialect().clauses.defaultInValues === false
+			? fail(new QueryBuilderDefect({ message: `${where}: ${currentDialect().name} has no per-cell DEFAULT in VALUES; group rows by their present columns` }), "NULL")
+			: "DEFAULT"
 		if (isExprLike(value)) return compileSqlFragment(value.toFragment())
 		const wire = encodeValue(table.columns[column]!.literalSchema, value, `${where}: ${context}, column ${column}`)
 		let name = `${VALUE_PARAM}${next++}`
@@ -1954,7 +1967,9 @@ function compileInsert(insert: CHInsert<any, any, any, any>, params: Record<stri
 			} else {
 				const selected = insertSelectSource(insert, selectQuery!, values)
 				columns = selected.columns
-				source = selected.sql
+				source = insert._state.conflict !== undefined && dialect.clauses.insertSelectNeedsWhere === true
+					? `SELECT * FROM (\n${selected.sql}\n) AS ${quoteIdent("__insert_source")} WHERE 1`
+					: selected.sql
 				readScope = selected.inner.tenantScope
 				const bound = tenantBoundOf(selected.inner)
 				readBound = bound

src/ch/dialect.ts

@@ -60,12 +60,24 @@ export interface DialectClauses {
 	/** Whether each `UNION ALL` branch is wrapped in parentheses. Postgres needs
 	 *  it for a branch with its own WITH, ORDER BY or LIMIT. */
 	readonly parenthesizeUnionBranches: boolean
+	/** SQLite preserves branch-local ORDER BY/LIMIT by selecting from each branch. */
+	readonly wrapUnionBranches?: boolean
+	/** SQLite requires LIMIT -1 before OFFSET when the caller omitted a limit. */
+	readonly offsetRequiresLimit?: boolean
 	/** `RETURNING` after an INSERT. Absent means no: an insert with
 	 *  `.returning()` fails to compile for the dialect. */
 	readonly returning?: boolean
 	/** `ON CONFLICT ... DO NOTHING / DO UPDATE` after an INSERT. Absent means
 	 *  no: an insert with `onConflict*` fails to compile for the dialect. */
 	readonly onConflict?: boolean
+	/** Whether ON CONFLICT can name a constraint rather than columns. */
+	readonly onConflictConstraint?: boolean
+	/** SQLite disambiguates INSERT SELECT ... ON CONFLICT with a WHERE. */
+	readonly insertSelectNeedsWhere?: boolean
+	/** Whether a VALUES tuple can contain DEFAULT for a missing cell. */
+	readonly defaultInValues?: boolean
+	/** Whether SELECT DISTINCT ON is supported. Undefined preserves existing dialects. */
+	readonly distinctOn?: boolean
 	/** `SETTINGS` on an INSERT, UPDATE or DELETE (ClickHouse). Absent means
 	 *  no: a write with `.settings()` fails to compile for the dialect. */
 	readonly writeSettings?: boolean
@@ -206,6 +218,7 @@ export const checkedLiteral = (dialect: Dialect, value: unknown, context: string
 
 const checkedSyntax = (dialect: Dialect): SqlSyntax => ({
 	dialect: dialect.name,
+	ilike: dialect.ilike,
 	quoteIdent: dialect.quoteIdent,
 	quoteString: (value) => checked(dialect, dialect.quoteString(value), "a string literal"),
 	literal: (value, context) => checkedLiteral(dialect, value, context),