Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1969a67
feat: add SQLite and D1 codecs and dialect profiles
artile Oct 24, 2026
ceb1f05
feat: preserve SQLite offset and union branch pagination
artile Oct 24, 2026
4fc495d
feat: reject SQL clauses SQLite cannot execute
artile Oct 24, 2026
f74a9db
fix: disambiguate SQLite insert-select upserts
artile Oct 24, 2026
80cf8a9
feat: add schema-aware SQLite aggregate and scalar functions
artile Oct 24, 2026
65e4bfa
test: verify SQLite transactions and local D1 safety checks
artile Oct 24, 2026
20b9478
docs: publish the SQLite entry and document its tested limits
artile Oct 24, 2026
0e96f31
docs: list SQLite under ON CONFLICT in the reference
artile Oct 25, 2026
bfe02f8
fix(sqlite): build external tables from their columns
artile Oct 25, 2026
f59dbc3
Merge remote-tracking branch 'origin/main' into fix/sqlite-review
Makisuo Oct 25, 2026
02e4857
fix(sqlite): portable LIKE escapes, no silent NUL truncation, exact f…
Makisuo Oct 25, 2026
0216fda
test(sqlite): run the core builder suite and every fuzz property on S…
Makisuo Oct 25, 2026
236a103
fix(sqlite): timestamps in SQLite's own text form, refuse integers pa…
Makisuo Oct 25, 2026
8184ac4
test(fuzz): fuzz unions and writes on every dialect
Makisuo Oct 25, 2026
7fcd3ca
fix(clickhouse): cast a union's mixed numeric column to Float64
Makisuo Oct 25, 2026
9ad1294
Merge branch 'test/fuzz-writes-unions' into feat/sqlite-d1-dialects
Makisuo Oct 25, 2026
f8b057a
test(fuzz): fuzz SQLite and D1 writes and unions
Makisuo Oct 25, 2026
79d0d72
test(fuzz): a query pinned to one org by its own filter is single-ten…
Makisuo Oct 25, 2026
34b3879
Merge branch 'test/fuzz-writes-unions' into feat/sqlite-d1-dialects
Makisuo Oct 25, 2026
4a7f6d3
fix(clickhouse): cast a union's numeric column around each branch, no…
Makisuo Oct 25, 2026
b0619ae
fix(clickhouse): order a union's mixed numeric column by exact keys, …
Makisuo Oct 25, 2026
77eec2a
fix(sqlite): refuse timestamps outside 0000-9999; fraction rule as a …
Makisuo Oct 25, 2026
04c6de3
fix(sqlite): refuse an out-of-range timestamp with a typed error, nev…
Makisuo Oct 25, 2026
6ca87b0
fix(clickhouse): order union numbers with CAST, which ClickHouse 26.2…
Makisuo Oct 25, 2026
1e9826d
Merge branch 'test/fuzz-writes-unions' into feat/sqlite-d1-dialects
Makisuo Oct 25, 2026
cfeaf59
test(fuzz): rerun ClickHouse's one-argument and() bug without the rew…
Makisuo Oct 25, 2026
2aa184f
test: retry a D1 request that never reached workerd
Makisuo Oct 25, 2026
4332136
test(fuzz): pin the and() server-bug shape ClickHouse 26.2 hit
Makisuo Oct 25, 2026
cff4474
Merge branch 'test/fuzz-writes-unions' into feat/sqlite-d1-dialects
Makisuo Oct 25, 2026
819937b
Merge remote-tracking branch 'origin/main' into feat/sqlite-d1-dialects
Makisuo Oct 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changeset/quiet-sqlite-dialects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
"@maple-dev/effect-orm": minor
---

Add the `/sqlite` query entry with SQLite codecs, functions and positional bindings. Use
`sqliteDialect` with an Effect SQLite SqlClient, or `d1Dialect` for Cloudflare D1's binding limit
and explicit refusal of interactive transactions. `timestamp` stores SQLite's own
`YYYY-MM-DD HH:MM:SS.SSS` UTC text, so it compares correctly with what `strftime` writes; `LIKE`
escapes with `\` as on every dialect; text with U+0000 and integers past 2^53 are refused rather
than silently changed. `table` is a query schema only: SQLite DDL and
migration generation are not implemented yet.
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@
> **Alpha.** The API is still changing and minor releases may break it. Pin an exact version,
> read the [changelog](./CHANGELOG.md) before upgrading, and expect rough edges.

Type-safe ClickHouse and Postgres queries, result decoding, and reproducible benchmarks for Effect and TypeScript.
Type-safe ClickHouse, Postgres and SQLite queries, result decoding, and reproducible benchmarks for Effect and TypeScript.

Formerly `@maple-dev/effect-clickhouse`.

SQLite and Cloudflare D1 use the same builder and row decoder through the new
[`/sqlite` entry](./docs/sqlite.md). SQLite schema/migration generation is not implemented.

[Read the documentation](https://effect-clickhouse.maple.dev) ·
[Getting started](./docs/getting-started.md) · [Recipes](./docs/recipes.md)

Expand Down
119 changes: 117 additions & 2 deletions bun.lock

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,7 @@ The optional `@maple-dev/effect-orm/benchmark` entry point and bundled
`ch-bench` CLI measure real queries, compare fixed workloads, and save evidence.
See [Benchmarking](./benchmarking.md) and the
[agent playbook](./benchmark-agent.md). The SQL builders remain driver-free.

## SQLite and D1

[SQLite and D1](./sqlite.md) covers `@maple-dev/effect-orm/sqlite`, its codecs and functions, Effect SQL drivers, tenant checks and transaction limits. SQLite migration generation is not implemented.
33 changes: 29 additions & 4 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ The package has one entry per database, each self-contained:
```ts
import * as CH from "@maple-dev/effect-orm/clickhouse"
import * as PG from "@maple-dev/effect-orm/postgres"
import * as SQLite from "@maple-dev/effect-orm/sqlite"
```

Both carry the whole [query builder](#query-builder-both-entries). Each adds its own column
All carry the whole [query builder](#query-builder-every-entry). Each adds its own column
types, functions, table definitions and a `compile` that defaults to its dialect:
[`/clickhouse`](#clickhouse) and [`/postgres`](#postgres). Everything else is on a
[subpath](#other-subpaths) for a narrower job.
Expand All @@ -32,9 +33,9 @@ uniformly, which some codebases prefer for exactly this reason.

---

## Query builder (both entries)
## Query builder (every entry)

Everything in this section is exported, identically, from both `/clickhouse` and `/postgres`.
Everything in this section is exported, identically, from `/clickhouse`, `/postgres` and `/sqlite`.

### Query construction

Expand All @@ -46,7 +47,7 @@ Everything in this section is exported, identically, from both `/clickhouse` and
| `unionAll` | `(...queries) => CHUnionQuery` |
| `update` | `(table) => CHUpdateStart`, then `CHUpdate`: `.set(record \| fn)`, `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)`. See [Updating and deleting](./updates-and-deletes.md) |
| `deleteFrom` | `(table) => CHDelete`: `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)` |
| `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) |
| `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres and SQLite). See [Inserting rows](./inserts.md) |

Tables come from the dialect's `table`: [`CH.table`](#tables-and-ddl) or
[`PG.table`](#tables-and-ddl-1).
Expand Down Expand Up @@ -530,3 +531,27 @@ verification, and JSON protocol.
Note `/sql` exports a `compile` (fragment → string) distinct from the dialect entries' `compile`
(query → `CompiledQuery`), and a `when` distinct from the query builder's `when` (optional
conditions).

## SQLite and D1

`@maple-dev/effect-orm/sqlite` exports the shared query builder and the four compile functions.
`sqliteDialect` is the default; `d1Dialect` uses the same SQL with a 100-binding limit and no
interactive transactions. See [SQLite and D1](./sqlite.md) for driver wiring and limits.

| Export | Purpose |
| --- | --- |
| `SqliteType` | A SQLite SQL type with its Effect codec |
| `text`, `integer`, `real` | Text, safe integer and finite number codecs |
| `boolean` | A boolean encoded as 0/1; other integers fail decoding |
| `timestamp` | An instant stored as SQLite's `YYYY-MM-DD HH:MM:SS.SSS` UTC text; reads ISO-8601 too |
| `sqliteTimestampText(epochMillis)` | That text for an instant, for raw SQL and seeds |
| `blob` | Bytes from a Uint8Array or D1's array wire representation |
| `json(schema)` | JSON text decoded through the supplied schema |
| `nullable(type)`, `custom`, `brand` | Type composition |
| `table(name, columns, options?)`, `TableOptions` | A query schema, including `tenantColumn`, defaults and computed-column metadata; no generated DDL |
| `table(name, ExternalTableDefinition)`, `ExternalTableDefinition` | `{ external: true, columns, tenantColumn? }`, the same query schema in the `/clickhouse` and `/postgres` form |
| `count`, `countDistinct`, `countIf`, `sum`, `avg`, `min`, `max` | SQLite aggregates; the last four can return SQL NULL |
| `lower`, `upper`, `length`, `coalesce` | SQLite scalar functions |

SQLite's table definition is for query typing and decoding. The schema/kit/migrate entries
currently support ClickHouse and Postgres DDL only.
133 changes: 133 additions & 0 deletions docs/sqlite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# SQLite and Cloudflare D1

Import `@maple-dev/effect-orm/sqlite` for the shared query builder, SQLite codecs and functions.
Compilation uses positional `?` bindings. Each occurrence of a parameter gets its own binding.
The package opens no connection: use the Effect `SqlClient` you already provide.

## Define and compile a query

```ts title="sqlite-quickstart.ts"
import * as SQLite from "@maple-dev/effect-orm/sqlite"

export const Acks = SQLite.table("acknowledgments", {
id: SQLite.text,
org_id: SQLite.text,
entry_id: SQLite.text,
}, { tenantColumn: "org_id" })

export const byOrg = SQLite.from(Acks)
.select("entry_id")
.where(($) => [$.org_id.eq(SQLite.param.string("orgId"))])
.orderBy(["entry_id", "asc"])

export const compiled = SQLite.compileUnsafe(byOrg, { orgId: "org_1" })
// compiled.parameters is ["org_1"]; compiled.tenantScope is "single-tenant".
```

`table` describes the query schema. Manage its actual DDL and migrations outside this package.
Unlike the ClickHouse and Postgres entries, SQLite has no schema diff, introspection, migration
ledger or kit dialect yet. A `table` value here cannot be passed to the DDL generators.

The entry exports `text`, `integer` (a safe JavaScript integer), `real` (finite numbers),
`boolean` (0/1), `blob`, `timestamp`, `json(schema)`, `nullable`, `custom` and `brand`.

`integer` refuses a stored value past 2^53 with a decode error: as a JavaScript number it would
no longer be the stored integer (D1 rounds it; node:sqlite refuses it). Use `custom` with a
bigint codec for such columns.

`timestamp` stores an instant as SQLite's own text form, `YYYY-MM-DD HH:MM:SS.SSS` in UTC, the
form `strftime('%Y-%m-%d %H:%M:%f')` and `datetime(..., 'subsec')` write. It reads that form
and ISO-8601; text without a zone is UTC. SQLite compares text by bytes, so a timestamp column
orders by time only when every value in it has one form: write them through the builder or with
`strftime('%Y-%m-%d %H:%M:%f', ...)`. `CURRENT_TIMESTAMP` writes no milliseconds
(`2026-01-01 00:00:00`), which compares as earlier than the same instant written with them.
`blob` accepts the Uint8Array wire form from Node and the array-of-bytes form from D1.
`json` always uses the supplied schema to validate decoded JSON.
Use `text` when a stored date/time must remain a string.

SQLite aggregates include `count`, `countDistinct`, `countIf`, `sum`, `avg`, `min` and `max`.
The last four return NULL over an empty input. Scalar helpers include `lower`, `upper`,
`length` and `coalesce`. SQLite's own text and case rules apply. Foreign built-in function
sets, such as ClickHouse's `count()`, fail compilation instead of producing invalid SQL.

## Execute on SQLite

Install a compatible Effect SQL driver, for example `@effect/sql-sqlite-node` on Node 24+.
No driver is a runtime dependency of this package. Use current SQLite with RETURNING support.

```ts title="sqlite-node.ts"
import * as SqliteClient from "@effect/sql-sqlite-node/SqliteClient"
import { Layer } from "effect"
import * as Db from "@maple-dev/effect-orm/database"
import { sqliteDialect } from "@maple-dev/effect-orm/sqlite"

export const SQLiteLive = Db.layerSqlClient({
dialect: sqliteDialect,
requireTenantScope: true,
}).pipe(Layer.provide(SqliteClient.layer({ filename: ":memory:" })))
```

`Db.run(query, params)` compiles for the database dialect and decodes the selected rows.
SQLite transactions and nested savepoints use the supplied client's `withTransaction`.
`serializable` is the only accepted explicit isolation level. Per-transaction access modes
and deferrable settings are refused. The driver must support the declared transaction behavior;
use the D1 profile for a D1 client.

## Execute on D1

Install `@effect/sql-d1` with a compatible Effect version. Provide the Worker's D1 binding:

```ts title="sqlite-d-one.ts"
import * as D1Client from "@effect/sql-d1/D1Client"
import { Layer } from "effect"
import * as Db from "@maple-dev/effect-orm/database"
import { d1Dialect } from "@maple-dev/effect-orm/sqlite"

export const d1DatabaseLayer = (db: D1Client.D1ClientConfig["db"]) =>
Db.layerSqlClient({ dialect: d1Dialect, requireTenantScope: true }).pipe(
Layer.provide(D1Client.layer({ db })),
)
```

Use `d1Dialect` when compiling a query separately as well:
`SQLite.compile(query, params, { dialect: SQLite.d1Dialect })`. A query compiled for `sqlite`
cannot later run on a database configured as `d1`; pass the builder to `Db.run` instead.

D1 compilation refuses more than [100 bound parameters per statement](https://developers.cloudflare.com/d1/platform/limits/).
It never splits a write implicitly. Chunk an import explicitly if needed; several chunks do
not make one atomic write. SQLite compilation uses its modern default limit of 32766; a driver
with a lower limit needs a custom dialect profile.

`Database.transaction` on D1 returns `TransactionUnsupported` before executing its body.
D1's [atomic batch API](https://developers.cloudflare.com/d1/worker-api/d1-database/#batch)
executes a fixed list of statements. It cannot run an arbitrary Effect transaction body with
reads that determine later writes. There is no ORM batch API in this first implementation.

## Safety and syntax limits

Derived schemas validate returned rows. A row with the wrong type returns
`CompiledQueryDecodeError`. Input values use each column's encoder before binding.

Tenant enforcement is **opt-in**, through `requireTenantScope: true`. A compiled query over a
tenant table must pin its `tenantColumn` to one value, or explicitly use `.crossTenant()` for
an administrative read/write. Raw SQL passed to `query` or `execute` has no tenant proof and
bypasses this guard. It is not a replacement for an application's authorization checks.

The dialect handles OFFSET without LIMIT, branch-local ordering/limits in UNION ALL, and
INSERT SELECT with ON CONFLICT. `LIKE` patterns escape `%`, `_` and `\` with `\` as on every
dialect: the builder writes `ESCAPE '\'`, which SQLite needs. A fraction is bound rather than
written as decimal text, because SQLite reads some decimals as a neighbouring double, and a
fraction compared with an `integer` column (`Amount > 9.75`) compiles as on Postgres. It refuses these unsupported forms at compile time:

- `DISTINCT ON`, row locks and `ILIKE`. Use `LIKE`, which on SQLite ignores ASCII case
(`'acme' LIKE 'ACME'` is true, unlike Postgres and ClickHouse).
- Text containing U+0000. SQLite stores it, but its drivers end text there, so the value would
read back cut short; the builder refuses it in literals and bound params.
- `ON CONSTRAINT` conflict targets. Name the indexed columns instead.
- Per-cell `DEFAULT` in a VALUES tuple. Omit a defaulted column from every row in the statement,
or group rows by their present columns. All-default rows are not supported by the builder.
- ClickHouse FORMAT/SETTINGS and built-in functions from another dialect.

The tests run real SQLite through `@effect/sql-sqlite-node` and local workerd/D1 through
Miniflare. They require no deployed database or Cloudflare credentials. Both run the shared core
builder suite (`tests/core.sqlite.test.ts`) and every fuzz property (`tests/fuzz`).
14 changes: 12 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
{
"name": "@maple-dev/effect-orm",
"version": "0.7.2",
"description": "Type-safe ClickHouse and Postgres queries, result decoding, and reproducible benchmarks for Effect and TypeScript",
"description": "Type-safe ClickHouse, Postgres and SQLite queries, result decoding, and reproducible benchmarks for Effect and TypeScript",
"keywords": [
"clickhouse",
"effect",
"orm",
"postgres",
"query-builder",
"sql",
"typescript"
"typescript",
"sqlite",
"cloudflare-d1"
],
"homepage": "https://effect-clickhouse.maple.dev",
"bugs": {
Expand Down Expand Up @@ -97,6 +99,11 @@
"types": "./dist/benchmark/cli.d.mts",
"import": "./dist/benchmark/cli.mjs",
"default": "./dist/benchmark/cli.mjs"
},
"./sqlite": {
"types": "./dist/sqlite.d.mts",
"import": "./dist/sqlite.mjs",
"default": "./dist/sqlite.mjs"
}
},
"publishConfig": {
Expand All @@ -121,13 +128,16 @@
"@effect/language-service": "^0.87.3",
"@effect/platform-bun": "4.0.0",
"@effect/sql-clickhouse": "4.0.0",
"@effect/sql-d1": "4.0.0",
"@effect/sql-pglite": "4.0.0",
"@effect/sql-sqlite-node": "4.0.0",
"@effect/vitest": "4.0.0",
"@electric-sql/pglite": "0.5.8",
"@types/node": "^22.10.2",
"effect": "4.0.0",
"expect-type": "^1.3.0",
"fast-check": "^4.9.0",
"miniflare": "4.20260730.0",
"tsdown": "^0.22.14",
"typescript": "^7.0.2",
"vitest": "^4.1.9"
Expand Down
4 changes: 2 additions & 2 deletions scripts/check-exports-documented.mjs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env node
// Every export of the two dialect entries is named in docs/reference.md, whose
// Every export of the three dialect entries is named in docs/reference.md, whose
// first line claims to be the catalog. A claim like that decays the moment
// anything is added, so it is checked rather than promised.
//
Expand All @@ -11,7 +11,7 @@ import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"

const root = join(dirname(fileURLToPath(import.meta.url)), "..")
const entries = ["clickhouse", "postgres"]
const entries = ["clickhouse", "postgres", "sqlite"]

const exportsOf = (entry) => {
const dts = join(root, `dist/${entry}.d.mts`)
Expand Down
Loading
Loading