Skip to content

feat: SQLite and Cloudflare D1 query dialects (#49, fuzzed) - #119

Merged
Makisuo merged 30 commits into
mainfrom
feat/sqlite-d1-dialects
Oct 25, 2026
Merged

Makisuo merged 30 commits into
mainfrom
feat/sqlite-d1-dialects

Conversation

@Makisuo

@Makisuo Makisuo commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

This is #49 by @tao-io, with review fixes on top. Their 9 commits are unchanged. GitHub doesn't let maintainers push to a fork owned by an organization, so the fixes couldn't go onto #49 directly. Refs #48.

What #49 adds

See #49: a /sqlite entry with codecs, functions, positional ? binding, sqliteDialect (client transactions and savepoints) and d1Dialect (100 bindings, no interactive transactions), plus compile-time refusals for what SQLite lacks.

What this adds

Fixes, found by running the fuzzer (#118) and the core builder suite on SQLite and D1:

  • LIKE escapes. SQLite has no default LIKE escape, so like("org\\_1") matched a literal backslash and never org_1. The dialect now writes ESCAPE '\', so \ escapes the same way on every dialect.
  • U+0000 is refused. SQLite drivers end text at NUL: node:sqlite reads 'a' || char(0) || 'b' back as 'a', whether bound or literal. feat: add SQLite and Cloudflare D1 query dialects #49's own NUL test failed on Node 24 (our mise.toml version) for this reason. The builder now refuses such a string at compile time instead of storing a value that reads back cut short.
  • Exact fractions. SQLite parses some decimals to a neighbouring double, both as a literal and through json_extract (1.23456789e-95). Fractions in expressions are now bound instead of written as text.
  • Amount > 9.75 compiles. It was refused because SQLite.integer only encodes whole numbers. It now behaves as on Postgres and ClickHouse; inserting a fraction is still refused.
  • Docs. docs/sqlite.md now states that SQLite's LIKE ignores ASCII case.

Tests:

  • tests/core.sqlite.test.ts runs every shared core case on node:sqlite and on a local D1 binding (Miniflare), with SQL snapshots for both. This is the "shared builder suite" criterion from MySQL / MariaDB dialect #53/Support SQLite and Cloudflare D1 through the existing dialect and Effect SQL APIs #48.
  • tests/fuzz/dialects/sqlite.ts registers sqlite and d1 with the fuzzer: query plans against the reference model, literal round-trips for every SQLite column type, compile invariants and hostile values.
  • New core cases pin LIKE escapes and LIKE case on every dialect. ILIKE gets its own case, so patterns-and-lists now runs on SQLite too.

Verification

  • bun run typecheck: pass.
  • bun run test with ClickHouse 26.8.2.7: 64 files, 1,219 tests passed, none skipped. Doc citations, export catalogs (151 /sqlite) and doc examples ok.
  • Every fuzz property at FUZZ_SCALE=20 on all six targets: 101/101 pass.
  • bun scripts/check-package.ts: ok.

Decisions

Second round: gaps closed (after review)

  • Timestamps in SQLite's own text form. SQLite.timestamp stored ISO text (…T16:28:54.750Z), while SQLite's own functions write YYYY-MM-DD HH:MM:SS. SQLite compares text by bytes, and ' ' sorts before 'T', so a row a DEFAULT had just stamped was not "later than an hour ago": wrong rows, no error.
    • It now stores YYYY-MM-DD HH:MM:SS.SSS in UTC, the form strftime('%f') and datetime(…,'subsec') write.
    • It reads that form and ISO; zoneless text is UTC.
    • Values outside years 0000–9999 are refused with a typed error. new Date(8.64e15) used to be written as garbage.
    • The literal fuzz builds its reference row with SQLite's datetime().
  • integer refuses values past 2^53. D1 rounds such an INTEGER to a different JS number, which then decoded silently. It is now a decode error.
  • The integer > 9.75 rule is a dialect flag (integerComparesFraction), so a custom profile spread from sqliteDialect keeps it.
  • SQLite and D1 now run the write and union fuzzers from test(fuzz): fuzz unions and writes on every dialect #122: INSERT/SELECT, ON CONFLICT, UPDATE, DELETE and RETURNING against a model, and per-cell DEFAULT refused.
  • The D1 test helper retries transport errors only (Miniflare "fetch failed" under a soak).
  • Changesets match the code (CodeRabbit on feat(sqlite): table DDL — columns, keys, indexes #50).
  • Verified against ClickHouse 26.2.19.43 and 26.8.2.7: bun run test (1,239 tests on this branch) and every fuzz property at FUZZ_SCALE=20.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

artile and others added 12 commits October 24, 2026 14:49
…ractions

Found by running the fuzzer and the core builder suite on SQLite and D1:

- LIKE has no default escape on SQLite, so `like("org\\_1")` matched a
  literal backslash and never `org_1`. The dialect now writes
  `ESCAPE '\'`, so `\` escapes `%`, `_` and `\` as on every dialect.
- SQLite drivers end text at U+0000: node:sqlite reads `'a' || char(0)`
  back as `'a'`, bound or literal. A string with U+0000 is refused at
  compile time instead of being stored and read back cut short. (The
  hex-cast literal kept NUL in the SQL, but not in the value.)
- SQLite parses some decimals to a neighbouring double, as a literal or
  through json_extract (`1.23456789e-95`). A fraction in an expression
  is bound instead of written as text; integers are still written.
- `Amount > 9.75` was refused because SQLite.integer only encodes whole
  numbers. A fraction compared with an integer column now compiles, as
  on Postgres and ClickHouse; inserting one is still refused.

docs/sqlite.md now states SQLite's ASCII-case-insensitive LIKE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…QLite and D1

- tests/core.sqlite.test.ts runs every shared core case on node:sqlite
  and on a local D1 binding (Miniflare), with SQL snapshots for both.
- tests/fuzz/dialects/sqlite.ts registers `sqlite` and `d1` with the
  fuzzer: query plans against the reference model, literal round-trips
  for every SQLite column type, compile invariants and hostile values.
  Their semantics: NULL first ascending, truncating integer division,
  `%` on integers, ASCII-case-insensitive LIKE.
- The real reference row is built from the double's mantissa and powers
  of two, since SQLite cannot parse some decimals exactly.
- New core cases pin LIKE escapes and case on every dialect; ILIKE moves
  to its own case so `patterns-and-lists` also runs on SQLite.
- Counterexamples pinned in tests/fuzz/regressions.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Makisuo and others added 2 commits October 25, 2026 18:41
…st 2^53

`SQLite.timestamp` stored and compared ISO text (`…T16:28:54.750Z`), while
SQLite's functions write `YYYY-MM-DD HH:MM:SS`. SQLite compares text by
bytes, and ' ' sorts before 'T', so a row a DEFAULT stamped just now was
not later than an hour ago: wrong rows, no error. Timestamps now use
SQLite's own form, `YYYY-MM-DD HH:MM:SS.SSS` in UTC (what strftime('%f')
and datetime(..., 'subsec') write), for inserts, comparisons and params;
both forms decode, and zoneless text is UTC. The literal fuzz's reference
row is now written by SQLite's datetime(), so builder values must equal
the database's own text. `sqliteTimestampText` is exported for raw SQL.

`SQLite.integer` decoded any integer-valued number. D1 rounds an INTEGER
past 2^53 to a different number, which then passed; it is a decode error
now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unions: `unionAll` of two or three row plans that select the same
aliases (each alias one sort in every branch), each branch with its own
join, filter, DISTINCT, ORDER BY and LIMIT, under the union's ORDER BY,
LIMIT and OFFSET. The model concatenates the branches' rows and pages
them; the union pages only when it orders by every alias, so ties are
equal rows. New properties `unions match the model (<target>, …)`.

Writes: a new suite. Each input gets a fresh scratch table on the
dialect's first target and runs a sequence of builder writes: multi-row
inserts leaving defaulted or nullable columns out, INSERT … SELECT,
ON CONFLICT DO NOTHING / DO UPDATE (with the existing and the excluded
row), UPDATE and DELETE with a filter or allRows(), and RETURNING. After
every write the table is read back and compared with a JS model of its
rows, and RETURNING with the rows the write touched. A write the dialect
does not support is still generated and must be refused at compile time.

`FuzzDialect.writes` declares the capabilities and the scratch table
(query `Table`, create/drop DDL, `exec`); `DialectModule` gains
unionAll, compileUnion, insertInto, update and deleteFrom. ClickHouse
targets pass mutations_sync=2 and lightweight_deletes_sync=2 so an
ALTER TABLE UPDATE or DELETE is visible to the next read.

Found: a ClickHouse 26 UNION ALL column whose branches have numeric
types with no common supertype (UInt64 and Int64, Int64 and Float64)
becomes a Variant, and ORDER BY on it fails. Recorded in known-issues
(clickhouse-union-variant-order-by) and pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Makisuo and others added 9 commits October 25, 2026 19:09
A UNION ALL column of numbers whose branches have no common ClickHouse
type (`Id + 1` is UInt64 and `Id - 1` Int64; Int64 and Float64) becomes a
Variant, and an ORDER BY on it fails ("Variant/Dynamic are not allowed in
ORDER BY keys"); with use_variant_as_common_type = 0 the union itself is
refused. The union fuzzer found it.

On a dialect with `unionNumberCast` (ClickHouse), a column is selected as
toFloat64(…) in every branch when every branch selects a number and their
types may differ: different declared types, or different computations.
Every numeric ClickHouse type decodes to a JS number, so Float64 loses
nothing the decode keeps. The known-issues entry is gone; the pinned
counterexample now passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Registers the write capabilities from #122 for sqlite and d1: every write
the dialect has (INSERT … SELECT, UPDATE, DELETE, ON CONFLICT, RETURNING),
and per-cell DEFAULT refused, as SQLite has none in VALUES. D1 compiles its
unions with the D1 dialect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ant for that org

The tenant oracle took "single-tenant" to mean the orgId param's tenant.
`OrgId IN ('org_2')` pins a query to org_2, which the builder rightly
reports as single-tenant; the oracle called reading org_2 a leak. Where
the plan's WHERE filters OrgId itself, any one org is right; otherwise
it must still be the param's. Found by a 20x soak on D1; pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t inside it

Inside a branch the cast ran before that branch's DISTINCT, so two UInt64
values past 2^53 (one Float64) became one row. The cast now wraps each
branch, SELECT toFloat64(x) AS x, … FROM (<branch>), so DISTINCT, GROUP BY
and LIMIT see exact values (CodeRabbit). The writes suite now accepts only
a QueryBuilderError or a QueryBuilderDefect as the refusal of a clause a
dialect lacks, not any defect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…never cast its values

The previous fix cast such a column to Float64, which an independent
review showed corrupts integers past 2^53 wherever the values stay on the
server: INSERT ... SELECT from the union stored 9007199254740992 for
9007199254740993, and an IN over it matched nothing. Only the union's
ORDER BY was ever broken (a Variant is not a valid sort key), so only the
sort key changes now: toFloat64(x), then accurateCastOrNull(x, 'Int128'),
which keeps integers past 2^53 in exact order. Checked on ClickHouse 26.8
with UInt64/Int64 values around 2^53 and 2^64 - 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dialect flag

From an independent review:
- A timestamp past year 9999 (new Date(8.64e15), a common max-date
  sentinel) was written as '+275760-09 13T00:00:00.', which then failed
  to decode, and `at > year 10000` bound text that sorts before every
  real timestamp, so it matched every row. Inserts, comparisons and
  params now refuse it at compile time; sqliteTimestampText throws.
- `integer > 9.75` keyed off the dialect name, so a profile spread from
  sqliteDialect under another name lost it. It is the dialect flag
  integerComparesFraction now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er a throw

The range check ran after a union member's encoder, which threw a
RangeError for a Date before year 0, so compile died instead of failing
with a QueryBuilderError (found by the hostile-values fuzz property).
Each member of the timestamp codec now checks the range on its decoded
side, and the encoder never throws; dateTimeLiteral fails with a typed
error too. Pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Makisuo and others added 7 commits October 25, 2026 19:46
… accepts on a Variant

toFloat64 refuses a Variant on ClickHouse 26.2 (CI); CAST(x AS
Nullable(Float64)) works on 26.2 and 26.8, Variant or not. Checked both
versions with the full suite and every fuzz property at 20x.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rite, rows still checked

The known-issue matcher for this ClickHouse server bug covered two
shapes; a 20x soak on 26.2 found a third (`(a OR 0 OR a) AND (a OR 0)`).
Rather than chase shapes, a ClickHouse target that gets exactly this
error runs the query again with optimize_extract_common_expressions = 0,
and its rows are compared with the model as usual: these inputs are now
checked, not skipped. The matcher and its helpers are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under a 20x soak Miniflare's local socket occasionally drops ('fetch
failed'). Like the ClickHouse helper, the D1 helper retries transport
errors only; a database error still fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fails with TOO_FEW_ARGUMENTS_FOR_FUNCTION if the rerun without the rewrite
is removed; passes, rows checked, with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Makisuo

Makisuo commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants