feat(insert): excluded and excludedAll for onConflictDoUpdate set

.changeset/upsert-excluded.md

@@ -0,0 +1,5 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+`excluded(columns)` and `excludedAll(except?)` build an `onConflictDoUpdate` `set` that copies columns from the row proposed for insertion: `set: excluded(["name", "rent"])` writes `SET "name" = "excluded"."name", "rent" = "excluded"."rent"`. `excludedAll` copies every column the insert writes, except the named ones and the conflict target's columns; a column the rows leave out keeps its old value. Names are checked against the table's writable columns. Setting the tenant column from the proposed row (here or in the callback form) now keeps the rows' tenant scope instead of making the insert cross-tenant. Postgres.

design/gap-review.md

@@ -44,8 +44,8 @@ builder; **P1** commonly used; **P2** niche.
   function names, which Postgres does not have); `select *`.
 - ClickHouse: FINAL, PREWHERE, LIMIT BY, SETTINGS on SELECT, the ARRAY JOIN clause.
 - Writes: UPDATE ... FROM with a typed VALUES source; ClickHouse `ALTER TABLE ... UPDATE/DELETE`
-  and lightweight DELETE; JSONEachRow bulk insert (`encodeInsertRows`); a set-all-from-`excluded`
-  helper; client-side default hooks like `$defaultFn` / `$onUpdate`.
+  and lightweight DELETE; JSONEachRow bulk insert (`encodeInsertRows`); ~~a set-all-from-`excluded`
+  helper~~ (built: `excluded` / `excludedAll`); client-side default hooks like `$defaultFn` / `$onUpdate`.
 - Runtime: `runFirst` / `runSingle` / `stream`; `observe` with duration, rows and error; replica
   routing for `route()`; a `SET LOCAL` helper for RLS; a mock `Database` for users' tests.
 - Types: select, insert and update row types and Schemas per table.

design/writes.md

@@ -171,6 +171,11 @@ Maple: 25 `DO UPDATE`, 43 `DO NOTHING`, 66 `excluded.` references, 6 `setWhere`.
   `ReplacingMergeTree`.
 - `DO NOTHING` with `RETURNING` returns no row for a skipped insert; the docs say so, because it
   is the usual Drizzle surprise.
+- `set: excluded(cols)` / `excludedAll(except)` are values, not callbacks: the compiler expands
+  them through the same proposed-row accessor the callback gets. That accessor is the one place a
+  dialect decides how to read the proposed row. SQLite writes `excluded."col"` as Postgres does.
+  MySQL writes `VALUES(col)` (or a row alias, 8.0.19+) and has no conflict target, so its
+  dialect will swap the accessor and refuse `target`; the helpers need no change.
 
 ## 7. ClickHouse specifics
 

docs/inserts.md

@@ -199,6 +199,44 @@ RETURNING "key" AS "key", "count" AS "count"
 - Calling either again replaces the clause. ClickHouse has no `ON CONFLICT` (deduplicate with a
   `ReplacingMergeTree` instead), so compiling one for it is a `QueryBuilderDefect`.
 
+### Set from the proposed row
+
+A sync upsert usually copies the new row over the old one. `excluded(columns)` and
+`excludedAll(except?)` build that `set`, so you do not list each column twice:
+
+```ts
+// Copy count only.
+PG.insertInto(Counters).values({ key: "a", count: 1 }).onConflictDoUpdate({ target: ["key"], set: PG.excluded(["count"]) })
+
+// Copy every column the rows write, but not locked.
+PG.insertInto(Counters)
+	.values({ key: "a", count: 1, locked: false })
+	.onConflictDoUpdate({ target: ["key"], set: PG.excludedAll(["locked"]) })
+```
+
+```sql
+INSERT INTO "counters" ("key", "count")
+VALUES ($1, $2)
+ON CONFLICT ("key") DO UPDATE SET "count" = "excluded"."count"
+
+INSERT INTO "counters" ("key", "count", "locked")
+VALUES ($1, $2, $3)
+ON CONFLICT ("key") DO UPDATE SET "count" = "excluded"."count"
+```
+
+- The names are checked against the table, as in a `set` record: a name that is not a column,
+  or a computed column, does not type-check.
+- `excludedAll` takes the columns the insert writes (the rows' keys, or the SELECT's aliases),
+  not every column of the table. A column the rows leave out keeps its old value; on a column
+  you do not write, `EXCLUDED` holds the default, and copying it would overwrite real data. It
+  also skips the target's columns, which are equal in both rows.
+- `col = excluded.col` on the tenant column keeps the rows' tenant scope: it writes the value
+  the rows already gave it.
+- To mix copied columns with other values (`updatedAt: now()`), use the callback form:
+  `set: ($, excluded) => ({ name: excluded.name, updatedAt: PG.now() })`.
+- The dialect writes the SQL that reads the proposed row, so the same `set` value works on any
+  dialect with an upsert.
+
 ## Settings
 
 On ClickHouse, `settings` adds a `SETTINGS` clause to the insert, before `VALUES` or the
@@ -242,6 +280,7 @@ A batch that mixes tenants on purpose, such as an ingest flush, declares it with
 | Compiling without `values`                        | `QueryBuilderDefect`                     |
 | `returning` for a dialect without RETURNING       | `QueryBuilderDefect`                     |
 | `onConflictDoUpdate` setting no or unknown columns | `QueryBuilderError` `InvalidArguments`  |
+| `excluded` / `excludedAll` naming an unknown column | `QueryBuilderError` `InvalidArguments` |
 | `onConflict*` without ON CONFLICT, a bad target   | `QueryBuilderDefect`                     |
 | `settings` without insert settings, a bad name    | `QueryBuilderDefect`                     |
 

docs/reference.md

@@ -47,6 +47,8 @@ Everything in this section is exported, identically, from both `/clickhouse` and
 | `update`     | `(table) => CHUpdateStart`, then `CHUpdate`: `.set(record \| fn)`, `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)`. See [Updating and deleting](./updates-and-deletes.md) |
 | `deleteFrom` | `(table) => CHDelete`: `.where(fn)` or `.allRows()`, `.returning(...)`, `.settings(record)` |
 | `insertInto` | `(table) => CHInsertStart`, then `CHInsert`; `.values(row \| rows)` or `.select(query)` sets its rows, `.settings(record)` ClickHouse `SETTINGS`, `.returning(...)` the RETURNING list, `.onConflictDoNothing(options?)` / `.onConflictDoUpdate(options)` the ON CONFLICT clause (Postgres). See [Inserting rows](./inserts.md) |
+| `excluded`    | `(columns) => ExcludedColumns`: an `onConflictDoUpdate` `set` that copies the named columns from the proposed row. See [On conflict](./inserts.md#on-conflict) |
+| `excludedAll` | `(except?) => ExcludedAll`: an `onConflictDoUpdate` `set` that copies every column the insert writes, except these and the target's |
 
 Tables come from the dialect's `table`: [`CH.table`](#tables-and-ddl) or
 [`PG.table`](#tables-and-ddl-1).
@@ -198,7 +200,7 @@ one), `Expr`, `ColumnRef`, `Condition`, `Comparable` (what a value of a type may
 against), `MapValueOf`, `Subquery`, `ParamMarker`, `ParamKind`, `CHQuery`, `CHUnionQuery`,
 `CHInsert`, `CHInsertStart`, `CHUpdate`, `CHUpdateStart`, `CHDelete`, `CHWrite`, `InsertValue`,
 `InsertSelectMisfits`, `InsertSelectMissing`, `InsertSettingValue`, `ConflictTarget`,
-`ConflictSet`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`,
+`ConflictSet`, `ExcludedSet`, `ExcludedColumns`, `ExcludedAll`, `OnConflictDoNothing`, `OnConflictDoUpdate`, `ColumnAccessor`,
 `JoinedColumnAccessor`, `JoinOnCallback`, `LockOptions`, `CompiledQuery`, `CompiledQueryInput`,
 `CompiledQueryRowSchema`, `RowSchemaMismatch`, `TenantScope`, `Dialect`, `DialectClauses`,
 `DialectTransactions`, `IsolationLevel`, `TransactionSettings`, `ParamStyle`, `FnResult`.

src/ch/compile.ts

@@ -11,7 +11,7 @@ import { compiled, compiledUnsafe, fail } from "./failure"
 import { columnSqlName, custom, dateTime, dateTime64, type CHType, type ColumnDefs } from "./types"
 import type { CHQuery, CHQueryState, NeedsSelect } from "./query"
 import type { CHUnionQuery } from "./union"
-import { isInsert, type AnyInsert, type CHInsert } from "./insert"
+import { isExcludedSet, isInsert, type AnyInsert, type CHInsert, type ConflictTarget, type ExcludedSet } from "./insert"
 import { isDelete, isUpdate, type CHDelete, type CHUpdate } from "./update"
 import type { Table } from "./table"
 import { createColumnAccessor, createQualifiedColumnAccessor, createJoinedColumnAccessor, sourceAlias } from "./query"
@@ -1670,6 +1670,8 @@ const onConflictClause = (
 	insert: CHInsert<any, any, any, any>,
 	cell: (column: string, value: unknown, context: string) => string,
 	wrote: (column: string, value: unknown, sql: string) => void,
+	/** The columns the INSERT writes, which `excludedAll` sets. */
+	columns: ReadonlyArray<string>,
 ): string => {
 	const { table, conflict } = insert._state
 	if (conflict === undefined) return ""
@@ -1706,15 +1708,70 @@ const onConflictClause = (
 	if (conflict.action === "nothing") return `\nON CONFLICT${target} DO NOTHING`
 
 	const existing = createQualifiedColumnAccessor(table.name, undefined, table.columns)
-	const excluded = createQualifiedColumnAccessor("excluded", undefined, table.columns)
-	const set = typeof conflict.set === "function" ? conflict.set(existing, excluded) : conflict.set
-	const assignments = setAssignments(table, set as Record<string, unknown>, cell, wrote, where, "onConflictDoUpdate")
+	const excluded = proposedRowAccessor(table)
+	const set =
+		typeof conflict.set === "function"
+			? conflict.set(existing, excluded)
+			: isExcludedSet(conflict.set)
+				? excludedSetRecord(table, conflict.set, excluded, columns, conflict.target, where)
+				: conflict.set
+	// `col = excluded.col` writes the value the rows already gave the column, so
+	// it adds nothing to the tenant evidence the rows hold.
+	const fromProposedRow = (column: string, value: unknown, sql: string) => {
+		if (value !== excluded[column]) wrote(column, value, sql)
+	}
+	const assignments = setAssignments(table, set as Record<string, unknown>, cell, fromProposedRow, where, "onConflictDoUpdate")
 	const condition = conflict.where?.(existing, excluded)
 	return `\nON CONFLICT${target} DO UPDATE SET ${assignments.join(", ")}${
 		condition === undefined ? "" : ` WHERE ${compileSqlFragment(condition.toFragment())}`
 	}`
 }
 
+/**
+ * The row proposed for insertion, as `onConflictDoUpdate`'s SET and WHERE
+ * read it. Postgres (and SQLite) write `excluded."col"`. A dialect that reads
+ * it otherwise, such as MySQL's `VALUES(col)`, changes only this accessor:
+ * the callback form and `excluded` / `excludedAll` all go through it.
+ */
+const proposedRowAccessor = (table: Table<string, ColumnDefs>) =>
+	createQualifiedColumnAccessor("excluded", undefined, table.columns)
+
+/**
+ * The SET record of `excluded(columns)` / `excludedAll(except)`. A column
+ * name that is not a column maps to `null`, so `setAssignments` reports it.
+ */
+const excludedSetRecord = (
+	table: Table<string, ColumnDefs>,
+	set: ExcludedSet,
+	excluded: Record<string, unknown>,
+	written: ReadonlyArray<string>,
+	target: ConflictTarget<any> | undefined,
+	where: string,
+): Record<string, unknown> => {
+	// Checked here, not in setAssignments: a mistyped exception would otherwise widen the SET.
+	const unknown = set._tag === "ExcludedAll" ? set.except.find((column) => !Object.hasOwn(table.columns, column)) : undefined
+	if (unknown !== undefined) {
+		return fail(new QueryBuilderError({
+			code: "InvalidArguments",
+			message: `${where}: excludedAll excepts ${JSON.stringify(unknown)}, which is not a column of the table`,
+		}), {})
+	}
+	const columns = set._tag === "ExcludedColumns" ? set.columns : allButSkipped(written, set.except, target)
+	return Object.fromEntries(
+		columns.map((column) => [column, Object.hasOwn(table.columns, column) ? excluded[column] : null]),
+	)
+}
+
+/** `excludedAll`'s columns. The target's are equal in both rows, so setting them changes nothing. */
+const allButSkipped = (
+	written: ReadonlyArray<string>,
+	except: ReadonlyArray<string>,
+	target: ConflictTarget<any> | undefined,
+): ReadonlyArray<string> => {
+	const skip = new Set<string>([...except, ...(target !== undefined && !("constraint" in target) ? target : [])])
+	return written.filter((column) => !skip.has(column))
+}
+
 /** The RETURNING clause and its row schema, or none without `returning`. */
 const returningOf = (
 	table: Table<string, ColumnDefs>,
@@ -1974,7 +2031,7 @@ function compileInsert(insert: CHInsert<any, any, any, any>, params: Record<stri
 			}
 			const conflictSql = onConflictClause(insert, cell, (column, value, sql) => {
 				if (column === tenant) pin(value, sql)
-			})
+			}, columns)
 			return [columns, source, conflictSql, readScope] as const
 		},
 	)

src/ch/insert.test-d.ts

@@ -138,6 +138,23 @@ CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ set:
 // @ts-expect-error a computed column cannot be set
 CH.insertInto(Spans).values({ OrgId: "o", Label: "l" }).onConflictDoUpdate({ target: ["OrgId"], set: { Day: "x" } })
 
+// excluded / excludedAll: SET from the proposed row, names checked against the table.
+const fromExcluded = CH.insertInto(Counters)
+	.values({ key: "k", count: 1 })
+	.onConflictDoUpdate({ target: ["key"], set: CH.excluded(["count"]) })
+// No params, so it compiles without any.
+expectTypeOf(PG.compileUnsafe(fromExcluded)).toEqualTypeOf<CH.CompiledQuery<never, undefined>>()
+CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll() })
+CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["count"]) })
+// @ts-expect-error not a column
+CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excluded(["nope"]) })
+// @ts-expect-error not a column
+CH.insertInto(Counters).values({ key: "k", count: 1 }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["nope"]) })
+// @ts-expect-error a computed column cannot be set
+CH.insertInto(Spans).values({ OrgId: "o", Label: "l" }).onConflictDoUpdate({ target: ["OrgId"], set: CH.excluded(["Day"]) })
+// @ts-expect-error at least one column
+CH.excluded([])
+
 // INSERT ... SELECT: the selected row must fit the table.
 const Daily = CH.table("daily", { OrgId: CH.string, Total: CH.uint64, Note: CH.nullable(CH.string) })
 const Source = CH.table("source", { OrgId: CH.string, Ms: CH.uint64, Label: CH.string })

src/ch/insert.test.ts

@@ -187,6 +187,71 @@ describe("insertInto", () => {
 			expect(compiled.parameters).toEqual(["o", "k", 1, false, true, false])
 		})
 
+		it("excluded(columns) sets each named column from the proposed row", () => {
+			const compiled = PG.compileUnsafe(
+				CH.insertInto(Counters).values(row).onConflictDoUpdate({ target: ["org", "key"], set: CH.excluded(["count", "locked"]) }),
+			)
+			expect(compiled.sql).toMatch(
+				/\nON CONFLICT \("org", "key"\) DO UPDATE SET "count" = "excluded"\."count", "locked" = "excluded"\."locked"$/,
+			)
+			expect(compiled.parameters).toEqual(["o", "k", 1, false])
+		})
+
+		it("excludedAll(except) sets each written column but the target's and the excepted ones", () => {
+			const upsert = (set: CH.ExcludedAll<"locked" | "count">) =>
+				PG.compileUnsafe(CH.insertInto(Counters).values(row).onConflictDoUpdate({ target: ["org", "key"], set })).sql
+			expect(upsert(CH.excludedAll())).toMatch(
+				/ DO UPDATE SET "count" = "excluded"\."count", "locked" = "excluded"\."locked"$/,
+			)
+			expect(upsert(CH.excludedAll(["locked"]))).toMatch(/ DO UPDATE SET "count" = "excluded"\."count"$/)
+			// A column the rows leave out is not set: its EXCLUDED value would be the default.
+			const keys = PG.compileUnsafe(
+				CH.insertInto(Keys)
+					.values({ id: "00000000-0000-0000-0000-000000000000", org_id: "o", name: "n", meta: {} })
+					.onConflictDoUpdate({ target: ["id"], set: CH.excludedAll(["org_id"]) }),
+			)
+			expect(keys.sql).toMatch(/ DO UPDATE SET "name" = "excluded"\."name", "meta" = "excluded"\."meta"$/)
+		})
+
+		it("setting the tenant column from the proposed row keeps the rows' tenant scope", () => {
+			const scope = (org: string) =>
+				PG.compileUnsafe(
+					CH.insertInto(Counters).values({ ...row, org }).onConflictDoUpdate({ target: ["key"], set: CH.excludedAll() }),
+				)
+			expect(scope("o").sql).toMatch(/ DO UPDATE SET "org" = "excluded"\."org", "count" = /)
+			expect(scope("o").tenantScope).toBe("single-tenant")
+			const twoTenants = PG.compileUnsafe(
+				CH.insertInto(Counters)
+					.values([row, { ...row, org: "p" }])
+					.onConflictDoUpdate({ target: ["key"], set: ($, excluded) => ({ org: excluded.org }) }),
+			)
+			expect(twoTenants.tenantScope).toBe("cross-tenant")
+		})
+
+		it.effect("excluded with an unknown column, or excludedAll that leaves none, fails", () =>
+			Effect.gen(function* () {
+				const insert = CH.insertInto(Counters).values(row)
+				const unknown = yield* Effect.flip(
+					PG.compile(insert.onConflictDoUpdate({ target: ["key"], set: CH.excluded(["nope"]) as any })),
+				)
+				expect(unknown.code).toBe("InvalidArguments")
+				expect(unknown.message).toContain('"nope"')
+				const unknownExcept = yield* Effect.flip(
+					PG.compile(insert.onConflictDoUpdate({ target: ["key"], set: CH.excludedAll(["nope"]) as any })),
+				)
+				expect(unknownExcept.code).toBe("InvalidArguments")
+				expect(unknownExcept.message).toContain('"nope"')
+				const none = yield* Effect.flip(
+					PG.compile(
+						CH.insertInto(Counters)
+							.values({ org: "o", key: "k", count: 1, locked: false })
+							.onConflictDoUpdate({ target: ["org", "key"], set: CH.excludedAll(["count", "locked"]) }),
+					),
+				)
+				expect(none.message).toContain("sets no columns")
+			}),
+		)
+
 		it("a SET that writes another tenant makes the insert cross-tenant", () => {
 			const insert = CH.insertInto(Counters).values(row)
 			const scope = (org: string) =>