diff --git a/.changeset/upsert-excluded.md b/.changeset/upsert-excluded.md new file mode 100644 index 0000000..8f4b146 --- /dev/null +++ b/.changeset/upsert-excluded.md @@ -0,0 +1,5 @@ +--- +"@maple-dev/effect-orm": minor +--- + +`excluded(columns)` and `excludedAll(except?)` build an `onConflictDoUpdate` `set` that copies columns from the row proposed for insertion: `set: excluded(["name", "rent"])` writes `SET "name" = "excluded"."name", "rent" = "excluded"."rent"`. `excludedAll` copies every column the insert writes, except the named ones and the conflict target's columns; a column the rows leave out keeps its old value. Names are checked against the table's writable columns. Setting the tenant column from the proposed row (here or in the callback form) now keeps the rows' tenant scope instead of making the insert cross-tenant. Postgres. diff --git a/design/gap-review.md b/design/gap-review.md index 852c5eb..e986bac 100644 --- a/design/gap-review.md +++ b/design/gap-review.md @@ -44,8 +44,8 @@ builder; **P1** commonly used; **P2** niche. function names, which Postgres does not have); `select *`. - ClickHouse: FINAL, PREWHERE, LIMIT BY, SETTINGS on SELECT, the ARRAY JOIN clause. - Writes: UPDATE ... FROM with a typed VALUES source; ClickHouse `ALTER TABLE ... UPDATE/DELETE` - and lightweight DELETE; JSONEachRow bulk insert (`encodeInsertRows`); a set-all-from-`excluded` - helper; client-side default hooks like `$defaultFn` / `$onUpdate`. + and lightweight DELETE; JSONEachRow bulk insert (`encodeInsertRows`); ~~a set-all-from-`excluded` + helper~~ (built: `excluded` / `excludedAll`); client-side default hooks like `$defaultFn` / `$onUpdate`. - Runtime: `runFirst` / `runSingle` / `stream`; `observe` with duration, rows and error; replica routing for `route()`; a `SET LOCAL` helper for RLS; a mock `Database` for users' tests. - Types: select, insert and update row types and Schemas per table. diff --git a/design/writes.md b/design/writes.md index abef65b..be89c1b 100644 --- a/design/writes.md +++ b/design/writes.md @@ -171,6 +171,11 @@ Maple: 25 `DO UPDATE`, 43 `DO NOTHING`, 66 `excluded.` references, 6 `setWhere`. `ReplacingMergeTree`. - `DO NOTHING` with `RETURNING` returns no row for a skipped insert; the docs say so, because it is the usual Drizzle surprise. +- `set: excluded(cols)` / `excludedAll(except)` are values, not callbacks: the compiler expands + them through the same proposed-row accessor the callback gets. That accessor is the one place a + dialect decides how to read the proposed row. SQLite writes `excluded."col"` as Postgres does. + MySQL writes `VALUES(col)` (or a row alias, 8.0.19+) and has no conflict target, so its + dialect will swap the accessor and refuse `target`; the helpers need no change. ## 7. ClickHouse specifics diff --git a/docs/inserts.md b/docs/inserts.md index 407ba91..1b7423e 100644 --- a/docs/inserts.md +++ b/docs/inserts.md @@ -199,6 +199,44 @@ RETURNING "key" AS "key", "count" AS "count" - Calling either again replaces the clause. ClickHouse has no `ON CONFLICT` (deduplicate with a `ReplacingMergeTree` instead), so compiling one for it is a `QueryBuilderDefect`. +### Set from the proposed row + +A sync upsert usually copies the new row over the old one. `excluded(columns)` and +`excludedAll(except?)` build that `set`, so you do not list each column twice: + +```ts +// Copy count only. +PG.insertInto(Counters).values({ key: "a", count: 1 }).onConflictDoUpdate({ target: ["key"], set: PG.excluded(["count"]) }) + +// Copy every column the rows write, but not locked. +PG.insertInto(Counters) + .values({ key: "a", count: 1, locked: false }) + .onConflictDoUpdate({ target: ["key"], set: PG.excludedAll(["locked"]) }) +``` + +```sql +INSERT INTO "counters" ("key", "count") +VALUES ($1, $2) +ON CONFLICT ("key") DO UPDATE SET "count" = "excluded"."count" + +INSERT INTO "counters" ("key", "count", "locked") +VALUES ($1, $2, $3) +ON CONFLICT ("key") DO UPDATE SET "count" = "excluded"."count" +``` + +- The names are checked against the table, as in a `set` record: a name that is not a column, + or a computed column, does not type-check. +- `excludedAll` takes the columns the insert writes (the rows' keys, or the SELECT's aliases), + not every column of the table. A column the rows leave out keeps its old value; on a column + you do not write, `EXCLUDED` holds the default, and copying it would overwrite real data. It + also skips the target's columns, which are equal in both rows. +- `col = excluded.col` on the tenant column keeps the rows' tenant scope: it writes the value + the rows already gave it. +- To mix copied columns with other values (`updatedAt: now()`), use the callback form: + `set: ($, excluded) => ({ name: excluded.name, updatedAt: PG.now() })`. +- The dialect writes the SQL that reads the proposed row, so the same `set` value works on any + dialect with an upsert. + ## Settings On ClickHouse, `settings` adds a `SETTINGS` clause to the insert, before `VALUES` or the @@ -242,6 +280,7 @@ A batch that mixes tenants on purpose, such as an ingest flush, declares it with | Compiling without `values` | `QueryBuilderDefect` | | `returning` for a dialect without RETURNING | `QueryBuilderDefect` | | `onConflictDoUpdate` setting no or unknown columns | `QueryBuilderError` `InvalidArguments` | +| `excluded` / `excludedAll` naming an unknown column | `QueryBuilderError` `InvalidArguments` | | `onConflict*` without ON CONFLICT, a bad target | `QueryBuilderDefect` | | `settings` without insert settings, a bad name | `QueryBuilderDefect` | diff --git a/docs/reference.md b/docs/reference.md index abc2d46..45d8a26 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -47,6 +47,8 @@ Everything in this section is exported, identically, from both `/clickhouse` and | `update` | `(table) => CHUpdateStart`, then `CHUpdate`: `.set(record \| fn)`, `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)`. See [Updating and deleting](./updates-and-deletes.md) | | `deleteFrom` | `(table) => CHDelete`: `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)` | | `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) | +| `excluded` | `(columns) => ExcludedColumns`: an `onConflictDoUpdate` `set` that copies the named columns from the proposed row. See [On conflict](./inserts.md#on-conflict) | +| `excludedAll` | `(except?) => ExcludedAll`: an `onConflictDoUpdate` `set` that copies every column the insert writes, except these and the target's | Tables come from the dialect's `table`: [`CH.table`](#tables-and-ddl) or [`PG.table`](#tables-and-ddl-1). @@ -198,7 +200,7 @@ one), `Expr`, `ColumnRef`, `Condition`, `Comparable` (what a value of a type may against), `MapValueOf`, `Subquery`, `ParamMarker`, `ParamKind`, `CHQuery`, `CHUnionQuery`, `CHInsert`, `CHInsertStart`, `CHUpdate`, `CHUpdateStart`, `CHDelete`, `CHWrite`, `InsertValue`, `InsertSelectMisfits`, `InsertSelectMissing`, `InsertSettingValue`, `ConflictTarget`, -`ConflictSet`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`, +`ConflictSet`, `ExcludedSet`, `ExcludedColumns`, `ExcludedAll`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`, `JoinedColumnAccessor`, `JoinOnCallback`, `LockOptions`, `CompiledQuery`, `CompiledQueryInput`, `CompiledQueryRowSchema`, `RowSchemaMismatch`, `TenantScope`, `Dialect`, `DialectClauses`, `DialectTransactions`, `IsolationLevel`, `TransactionSettings`, `ParamStyle`, `FnResult`. diff --git a/src/ch/compile.ts b/src/ch/compile.ts index 0aaaac1..829d247 100644 --- a/src/ch/compile.ts +++ b/src/ch/compile.ts @@ -11,7 +11,7 @@ import { compiled, compiledUnsafe, fail } from "./failure" import { columnSqlName, custom, dateTime, dateTime64, type CHType, type ColumnDefs } from "./types" import type { CHQuery, CHQueryState, NeedsSelect } from "./query" import type { CHUnionQuery } from "./union" -import { isInsert, type AnyInsert, type CHInsert } from "./insert" +import { isExcludedSet, isInsert, type AnyInsert, type CHInsert, type ConflictTarget, type ExcludedSet } from "./insert" import { isDelete, isUpdate, type CHDelete, type CHUpdate } from "./update" import type { Table } from "./table" import { createColumnAccessor, createQualifiedColumnAccessor, createJoinedColumnAccessor, sourceAlias } from "./query" @@ -1670,6 +1670,8 @@ const onConflictClause = ( insert: CHInsert, cell: (column: string, value: unknown, context: string) => string, wrote: (column: string, value: unknown, sql: string) => void, + /** The columns the INSERT writes, which `excludedAll` sets. */ + columns: ReadonlyArray, ): string => { const { table, conflict } = insert._state if (conflict === undefined) return "" @@ -1706,15 +1708,70 @@ const onConflictClause = ( if (conflict.action === "nothing") return `\nON CONFLICT${target} DO NOTHING` const existing = createQualifiedColumnAccessor(table.name, undefined, table.columns) - const excluded = createQualifiedColumnAccessor("excluded", undefined, table.columns) - const set = typeof conflict.set === "function" ? conflict.set(existing, excluded) : conflict.set - const assignments = setAssignments(table, set as Record, cell, wrote, where, "onConflictDoUpdate") + const excluded = proposedRowAccessor(table) + const set = + typeof conflict.set === "function" + ? conflict.set(existing, excluded) + : isExcludedSet(conflict.set) + ? excludedSetRecord(table, conflict.set, excluded, columns, conflict.target, where) + : conflict.set + // `col = excluded.col` writes the value the rows already gave the column, so + // it adds nothing to the tenant evidence the rows hold. + const fromProposedRow = (column: string, value: unknown, sql: string) => { + if (value !== excluded[column]) wrote(column, value, sql) + } + const assignments = setAssignments(table, set as Record, cell, fromProposedRow, where, "onConflictDoUpdate") const condition = conflict.where?.(existing, excluded) return `\nON CONFLICT${target} DO UPDATE SET ${assignments.join(", ")}${ condition === undefined ? "" : ` WHERE ${compileSqlFragment(condition.toFragment())}` }` } +/** + * The row proposed for insertion, as `onConflictDoUpdate`'s SET and WHERE + * read it. Postgres (and SQLite) write `excluded."col"`. A dialect that reads + * it otherwise, such as MySQL's `VALUES(col)`, changes only this accessor: + * the callback form and `excluded` / `excludedAll` all go through it. + */ +const proposedRowAccessor = (table: Table) => + createQualifiedColumnAccessor("excluded", undefined, table.columns) + +/** + * The SET record of `excluded(columns)` / `excludedAll(except)`. A column + * name that is not a column maps to `null`, so `setAssignments` reports it. + */ +const excludedSetRecord = ( + table: Table, + set: ExcludedSet, + excluded: Record, + written: ReadonlyArray, + target: ConflictTarget | undefined, + where: string, +): Record => { + // Checked here, not in setAssignments: a mistyped exception would otherwise widen the SET. + const unknown = set._tag === "ExcludedAll" ? set.except.find((column) => !Object.hasOwn(table.columns, column)) : undefined + if (unknown !== undefined) { + return fail(new QueryBuilderError({ + code: "InvalidArguments", + message: `${where}: excludedAll excepts ${JSON.stringify(unknown)}, which is not a column of the table`, + }), {}) + } + const columns = set._tag === "ExcludedColumns" ? set.columns : allButSkipped(written, set.except, target) + return Object.fromEntries( + columns.map((column) => [column, Object.hasOwn(table.columns, column) ? excluded[column] : null]), + ) +} + +/** `excludedAll`'s columns. The target's are equal in both rows, so setting them changes nothing. */ +const allButSkipped = ( + written: ReadonlyArray, + except: ReadonlyArray, + target: ConflictTarget | undefined, +): ReadonlyArray => { + const skip = new Set([...except, ...(target !== undefined && !("constraint" in target) ? target : [])]) + return written.filter((column) => !skip.has(column)) +} + /** The RETURNING clause and its row schema, or none without `returning`. */ const returningOf = ( table: Table, @@ -1974,7 +2031,7 @@ function compileInsert(insert: CHInsert, params: Record { if (column === tenant) pin(value, sql) - }) + }, columns) return [columns, source, conflictSql, readScope] as const }, ) diff --git a/src/ch/insert.test-d.ts b/src/ch/insert.test-d.ts index 54cb51f..68d24ab 100644 --- a/src/ch/insert.test-d.ts +++ b/src/ch/insert.test-d.ts @@ -138,6 +138,23 @@ CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ set: // @ts-expect-error a computed column cannot be set CH.insertInto(Spans).values({ OrgId: "o", Label: "l" }).onConflictDoUpdate({ target: ["OrgId"], set: { Day: "x" } }) +// excluded / excludedAll: SET from the proposed row, names checked against the table. +const fromExcluded = CH.insertInto(Counters) + .values({ key: "k", count: 1 }) + .onConflictDoUpdate({ target: ["key"], set: CH.excluded(["count"]) }) +// No params, so it compiles without any. +expectTypeOf(PG.compileUnsafe(fromExcluded)).toEqualTypeOf>() +CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll() }) +CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["count"]) }) +// @ts-expect-error not a column +CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excluded(["nope"]) }) +// @ts-expect-error not a column +CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["nope"]) }) +// @ts-expect-error a computed column cannot be set +CH.insertInto(Spans).values({ OrgId: "o", Label: "l" }).onConflictDoUpdate({ target: ["OrgId"], set: CH.excluded(["Day"]) }) +// @ts-expect-error at least one column +CH.excluded([]) + // INSERT ... SELECT: the selected row must fit the table. const Daily = CH.table("daily", { OrgId: CH.string, Total: CH.uint64, Note: CH.nullable(CH.string) }) const Source = CH.table("source", { OrgId: CH.string, Ms: CH.uint64, Label: CH.string }) diff --git a/src/ch/insert.test.ts b/src/ch/insert.test.ts index 7017ed1..a1c50e6 100644 --- a/src/ch/insert.test.ts +++ b/src/ch/insert.test.ts @@ -187,6 +187,71 @@ describe("insertInto", () => { expect(compiled.parameters).toEqual(["o", "k", 1, false, true, false]) }) + it("excluded(columns) sets each named column from the proposed row", () => { + const compiled = PG.compileUnsafe( + CH.insertInto(Counters).values(row).onConflictDoUpdate({ target: ["org", "key"], set: CH.excluded(["count", "locked"]) }), + ) + expect(compiled.sql).toMatch( + /\nON CONFLICT \("org", "key"\) DO UPDATE SET "count" = "excluded"\."count", "locked" = "excluded"\."locked"$/, + ) + expect(compiled.parameters).toEqual(["o", "k", 1, false]) + }) + + it("excludedAll(except) sets each written column but the target's and the excepted ones", () => { + const upsert = (set: CH.ExcludedAll<"locked" | "count">) => + PG.compileUnsafe(CH.insertInto(Counters).values(row).onConflictDoUpdate({ target: ["org", "key"], set })).sql + expect(upsert(CH.excludedAll())).toMatch( + / DO UPDATE SET "count" = "excluded"\."count", "locked" = "excluded"\."locked"$/, + ) + expect(upsert(CH.excludedAll(["locked"]))).toMatch(/ DO UPDATE SET "count" = "excluded"\."count"$/) + // A column the rows leave out is not set: its EXCLUDED value would be the default. + const keys = PG.compileUnsafe( + CH.insertInto(Keys) + .values({ id: "00000000-0000-0000-0000-000000000000", org_id: "o", name: "n", meta: {} }) + .onConflictDoUpdate({ target: ["id"], set: CH.excludedAll(["org_id"]) }), + ) + expect(keys.sql).toMatch(/ DO UPDATE SET "name" = "excluded"\."name", "meta" = "excluded"\."meta"$/) + }) + + it("setting the tenant column from the proposed row keeps the rows' tenant scope", () => { + const scope = (org: string) => + PG.compileUnsafe( + CH.insertInto(Counters).values({ ...row, org }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll() }), + ) + expect(scope("o").sql).toMatch(/ DO UPDATE SET "org" = "excluded"\."org", "count" = /) + expect(scope("o").tenantScope).toBe("single-tenant") + const twoTenants = PG.compileUnsafe( + CH.insertInto(Counters) + .values([row, { ...row, org: "p" }]) + .onConflictDoUpdate({ target: ["key"], set: ($, excluded) => ({ org: excluded.org }) }), + ) + expect(twoTenants.tenantScope).toBe("cross-tenant") + }) + + it.effect("excluded with an unknown column, or excludedAll that leaves none, fails", () => + Effect.gen(function* () { + const insert = CH.insertInto(Counters).values(row) + const unknown = yield* Effect.flip( + PG.compile(insert.onConflictDoUpdate({ target: ["key"], set: CH.excluded(["nope"]) as any })), + ) + expect(unknown.code).toBe("InvalidArguments") + expect(unknown.message).toContain('"nope"') + const unknownExcept = yield* Effect.flip( + PG.compile(insert.onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["nope"]) as any })), + ) + expect(unknownExcept.code).toBe("InvalidArguments") + expect(unknownExcept.message).toContain('"nope"') + const none = yield* Effect.flip( + PG.compile( + CH.insertInto(Counters) + .values({ org: "o", key: "k", count: 1, locked: false }) + .onConflictDoUpdate({ target: ["org", "key"], set: CH.excludedAll(["count", "locked"]) }), + ), + ) + expect(none.message).toContain("sets no columns") + }), + ) + it("a SET that writes another tenant makes the insert cross-tenant", () => { const insert = CH.insertInto(Counters).values(row) const scope = (org: string) => diff --git a/src/ch/insert.ts b/src/ch/insert.ts index bf6adc0..b42bd6d 100644 --- a/src/ch/insert.ts +++ b/src/ch/insert.ts @@ -128,6 +128,56 @@ export type ConflictSet>]?: InsertValue | undefined } +/** + * A SET built from the row proposed for insertion, which `onConflictDoUpdate` + * takes as `set`: `excluded(columns)` names the columns, `excludedAll(except)` + * takes every column the insert writes. The dialect picks the SQL that reads + * the proposed row, so one value works on every dialect with an upsert. + */ +export type ExcludedSet = ExcludedColumns | ExcludedAll + +/** `excluded(columns)`: `SET col = EXCLUDED.col` for each named column. */ +export interface ExcludedColumns { + readonly _tag: "ExcludedColumns" + readonly columns: ReadonlyArray +} + +/** + * `excludedAll(except)`: `SET col = EXCLUDED.col` for each column the insert + * writes, except the named ones and the conflict target's columns. + */ +export interface ExcludedAll { + readonly _tag: "ExcludedAll" + readonly except: ReadonlyArray +} + +/** + * Set the named columns from the row proposed for insertion: + * `onConflictDoUpdate({ target: ["key"], set: excluded(["count", "name"]) })`. + * The names are checked against the insert's table where `set` takes them. + */ +export const excluded = (columns: readonly [K, ...Array]): ExcludedColumns => ({ + _tag: "ExcludedColumns", + columns: [...columns], +}) + +/** + * Set every column the insert writes from the row proposed for insertion, + * except the named columns and the conflict target's columns: + * `onConflictDoUpdate({ target: ["id"], set: excludedAll(["createdAt"]) })`. + */ +export const excludedAll = (except: ReadonlyArray = []): ExcludedAll => ({ + _tag: "ExcludedAll", + except: [...except], +}) + +/** @internal */ +export const isExcludedSet = (set: unknown): set is ExcludedSet => + typeof set === "object" && + set !== null && + "_tag" in set && + (set._tag === "ExcludedColumns" || set._tag === "ExcludedAll") + export interface OnConflictDoNothing { /** Omit to skip a row that conflicts on any unique index or constraint. */ readonly target?: ConflictTarget @@ -164,9 +214,12 @@ export interface OnConflictDoUpdate< /** * The columns to write into the existing row. As a callback, `$` is the * existing row and `excluded` the row that was proposed for insertion: - * `(($, excluded) => ({ count: $.count.add(excluded.count) }))`. + * `(($, excluded) => ({ count: $.count.add(excluded.count) }))`. Or + * `excluded(columns)` / `excludedAll(except)`, to copy columns from the + * proposed row. */ readonly set: + | ExcludedSet> | (S & OnlyColumns>) | (($: ColumnAccessor, excluded: ColumnAccessor) => S & OnlyColumns>) /** Update only the existing rows this holds for; the others are skipped. */ diff --git a/src/core.ts b/src/core.ts index b32adcf..9ed2459 100644 --- a/src/core.ts +++ b/src/core.ts @@ -111,6 +111,9 @@ export { type CHInsertStart, type ConflictSet, type ConflictTarget, + type ExcludedAll, + type ExcludedColumns, + type ExcludedSet, type InsertRow, type InsertRowOf, type InsertSelectMisfits, @@ -119,6 +122,8 @@ export { type InsertValue, type OnConflictDoNothing, type OnConflictDoUpdate, + excluded, + excludedAll, insertInto, } from "./ch/insert" diff --git a/src/database/database.test.ts b/src/database/database.test.ts index 072673f..cbc6451 100644 --- a/src/database/database.test.ts +++ b/src/database/database.test.ts @@ -456,6 +456,33 @@ layer(Live, { excludeTestServices: true })("Database on PGlite", (it) => { }), ) + it.effect("excluded and excludedAll copy the proposed row into the existing one", () => + Effect.gen(function* () { + yield* Db.execute( + Db.sql`CREATE TABLE units (id text PRIMARY KEY, name text NOT NULL, rent int4 NOT NULL, created text NOT NULL DEFAULT 'seed')`, + ) + const Units = CH.table("units", { id: PG.text, name: PG.text, rent: PG.int4, created: PG.text }, { defaults: ["created"] }) + yield* Db.run(CH.insertInto(Units).values({ id: "u1", name: "A", rent: 1, created: "first" })) + yield* Db.run( + CH.insertInto(Units) + .values({ id: "u1", name: "B", rent: 2, created: "second" }) + .onConflictDoUpdate({ target: ["id"], set: CH.excluded(["name"]) }), + ) + expect(yield* Db.run(CH.from(Units).select("name", "rent", "created"))).toEqual([{ name: "B", rent: 1, created: "first" }]) + // The rows leave `created` out, so excludedAll keeps it rather than writing the default. + const synced = yield* Db.run( + CH.insertInto(Units) + .values([{ id: "u1", name: "C", rent: 3 }, { id: "u2", name: "D", rent: 4 }]) + .onConflictDoUpdate({ target: ["id"], set: CH.excludedAll() }) + .returning("id", "name", "rent", "created"), + ) + expect([...synced].sort((x, y) => x.id.localeCompare(y.id))).toEqual([ + { id: "u1", name: "C", rent: 3, created: "first" }, + { id: "u2", name: "D", rent: 4, created: "seed" }, + ]) + }), + ) + it.effect("insert ... select copies rows, with ON CONFLICT and RETURNING", () => Effect.gen(function* () { yield* Db.execute(Db.sql`CREATE TABLE src (org text NOT NULL, n int4 NOT NULL)`)