Skip to content

fix(parser)!: treat a GO line inside a SQL Server statement as an error - #868

Merged
ajitpratap0 merged 125 commits into
mainfrom
fix/tsql-go-inside-statement
Oct 26, 2026
Merged

ajitpratap0 merged 125 commits into
mainfrom
fix/tsql-go-inside-statement

Conversation

@ajitpratap0

Copy link
Copy Markdown
Owner

Summary

Part 3 of 3 splitting #858. It stacks on #867. Once this is merged, the combined tree is exactly #858's head c8bd83b7; only the CHANGELOG wording differs.

sqlcmd ends a batch at any line holding only GO [n], wherever that line appears. On main the parser read such a line as an identifier. SELECT a,\nGO\nFROM t became the column list a, go and formatted as SELECT a, go FROM t, which sqlcmd would split somewhere else. With this PR, in SQL Server:

  • A statement that runs across a GO [n] line fails with "GO batch separator inside a statement: a line holding only GO ends the batch".
  • GO 0, GO 1.5 or an overflowing count alone on a line fails with "invalid GO repeat count", instead of a FROM / end-of-statement error from the statement before it.
  • Recovery parsing reports the error and resumes at the GO line, so the next batch is still parsed.

The check is SQL Server only, and the 4 restored tests cover parsing, recovery and FormatWithDialect.

Test Plan

  • task check: exit 0
  • TestParser_GOLineInsideStatement, TestParser_RecoveryStopsAtGOLineInsideStatement, TestBatchSeparatorInvalidCountMessage, TestBatchSeparatorInsideStatementIsAnError pass
  • TestRealWorldCorpus: 0 known failures

Breaking Changes

SQL Server input with a line holding only GO [n] inside a statement no longer parses. Main accepted it with go taken as a column or alias name. Other dialects are unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

Ajit Pratap Singh and others added 30 commits October 22, 2026 17:46
EXPLAIN SELECT ... discarded the query and formatted as DESCRIBE "SELECT";
EXPLAIN ANALYZE|VERBOSE <stmt> split into DESCRIBE ANALYZE; <stmt>, so
formatting turned a non-executing EXPLAIN into the executable statement.

Add ast.ExplainStatement {Keyword, Options, Statement} covering EXPLAIN,
DESCRIBE and DESC with options (ANALYZE, (FORMAT JSON), QUERY PLAN,
FORMAT=JSON, PIPELINE, PLAN FOR), render it in the formatter, and let the
CLI format it instead of refusing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…ith ratchet

Adds testdata/realworld/<dialect>/ corpora (PostgreSQL, MySQL, MariaDB,
SQL Server, Oracle, SQLite, Snowflake, ClickHouse) of production-style app,
reporting, ETL, DDL and dialect-specific SQL, plus:

- TestRealWorldCorpus: split -> parse -> format -> reparse -> idempotency
  -> dropped-word check for every statement, ratcheted against
  testdata/realworld/known_failures.txt (-update-known-failures regenerates)
- scripts/dialect-audit: CLI harness for the same pipeline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…IME ZONE, COLLATE, bitwise, ESCAPE)

Close the operator and predicate gaps found by the real-world corpus.

Parser
- IS [NOT] TRUE/FALSE/UNKNOWN, IS [NOT] DISTINCT FROM, SQLite "a IS [NOT] expr",
  PostgreSQL/SQLite ISNULL/NOTNULL (normalised to IS [NOT] NULL).
- = ANY/ALL/SOME over arrays and placeholders (new AnyAllExpression); the
  subquery forms keep AnyExpression/AllExpression.
- Postfix AT TIME ZONE and COLLATE (new AtTimeZoneExpression and
  CollateExpression), binding tighter than arithmetic; COLLATE on CREATE
  INDEX columns fills IndexColumn.Collate.
- Binary operators @@, &&, &, |, #, <<, >>, <->, <=>, == via a precedence
  climbing level between comparison and ||, unary ~ and MySQL !, MySQL
  DIV/MOD/XOR/SOUNDS LIKE/&&/:=, SQLite GLOB/MATCH, SIMILAR TO, NOT REGEXP/
  RLIKE. Word operators are gated to the dialect that defines them. <=>, <<,
  >>, <->, == are read from adjacent tokens until the tokenizer emits them.
- LIKE/ILIKE/SIMILAR TO ... ESCAPE (BinaryExpression.Escape).
- Predicates and comparisons chain left to right (a IN (1, 2) = TRUE); SQLite
  IN () and IN tbl; ClickHouse GLOBAL IN keeps the GLOBAL flag; ClickHouse
  cond ? a : b (TernaryExpression).
- Data types: array dimensions (int[3][4]), schema-qualified, multi-word
  (DOUBLE PRECISION, CHARACTER VARYING, TIMESTAMP WITH TIME ZONE, SIGNED
  INTEGER) and CHAR(n) CHARACTER SET x, shared by CAST and ::.
- AND and OR are stored in canonical upper case.

Formatter and AST
- Render the new nodes and fields; extend the operator and data-type
  allow-lists so they are written verbatim.
- format_sql (MCP) now upper-cases AND/OR by default; tests and docs updated.

Real-world corpus: 59 statements now pass, 6 move to a later failure stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… and MERGE

UPDATE and DELETE read a single table name and then expected SET/WHERE, and
the MySQL row limit was parsed but had no AST field, so the formatter
silently dropped LIMIT and the reformatted statement touched every row.

UPDATE / DELETE:
- targets with AS / bare aliases, PostgreSQL ONLY, Oracle derived-table
  targets and FROM-less DELETE, SQL Server TOP (n) [PERCENT]
- MySQL/MariaDB multi-table forms (UPDATE t JOIN u / UPDATE t1, t2,
  DELETE t1[, t2] FROM ..., DELETE FROM t1, t2 USING ...), SQL Server
  DELETE t FROM ... and UPDATE ... FROM t JOIN u
- DELETE ... USING routed into DeleteStatement.Using
- statement modifiers (LOW_PRIORITY, QUICK, IGNORE; SQLite OR ...)
- ORDER BY ... LIMIT n stored on the AST and rendered
- SET targets: qualified columns, (a, b) tuples, @variables, DEFAULT values
- SQL Server OUTPUT [INTO], RETURNING aliases and Oracle RETURNING ... INTO

INSERT: MySQL/SQLite/Snowflake modifiers, INTO-less SQL Server/MySQL form,
VALUES () and DEFAULT items, MySQL row alias, OVERRIDING SYSTEM|USER VALUE,
ON CONFLICT expression targets and partial-index WHERE, OUTPUT ... INTO,
ClickHouse SETTINGS / FUNCTION / FORMAT (FORMAT was dropped before), Oracle
INSERT ALL / INSERT FIRST. REPLACE gains SELECT, SET and RETURNING forms.

MERGE: derived-table sources, DO NOTHING, NOT MATCHED BY TARGET, RETURNING,
Oracle UPDATE ... WHERE / DELETE WHERE and INSERT ... WHERE; the formatter
now renders the OUTPUT clause it used to drop.

New dialect fields live in embedded structs (ast.DMLTargetFields,
ast.InsertDialectFields) so single-table code that reads TableName/Alias is
unchanged; keyword fields are allow-listed on render and reset by the pool
release functions. 91 real-world corpus statements now pass (one moves from
parse to the pre-existing INTERVAL drop).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
parseStatement was a closed switch, so BEGIN, COMMIT, SET, USE, CALL,
GRANT, SHOW ... LIKE, T-SQL DECLARE/EXEC/IF and the rest of the utility
family failed with "expected statement". Backtick- and bracket-quoted
keywords lost their quoting (`key` became the KEY token, [Group] formatted
as a bare GROUP), and keywords such as share, filter and owner were
rejected as aliases or columns.

- Add ast.TransactionStatement, SetStatement, UseStatement, CallStatement
  and a verbatim UtilityStatement whose SQL() reproduces the statement,
  rebuilt from its tokens with delimiters and escapes restored. Dispatch
  them by value after the typed statements, only at a statement boundary
  so the tail of a half-parsed statement is never split off. BEGIN/END
  are gated per dialect (T-SQL and PL/SQL blocks, END only in PostgreSQL
  and SQLite).
- SHOW forms ShowStatement cannot model, Snowflake USE and stage
  operations are now verbatim statements that re-parse (they formatted
  as an "-- UNSUPPORTED" comment); MariaDB ANALYZE <stmt> is an
  ExplainStatement.
- DescribeStatement keeps DESC vs DESCRIBE and the object kind; PRAGMA
  accepts a schema-qualified name.
- Delimited identifiers are never remapped to keywords; Identifier gains
  QuoteStyle so SQL() re-quotes with the source delimiters (`from`,
  [Group]); @/@@ variables are never quoted.
- Accept keyword words as aliases after AS and the role-option keywords
  (OWNER, URL, ...) as column names; allow a delimited implicit alias.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
String literals no longer corrupt on format. The tokenizer and the renderer
disagreed about backslashes: PostgreSQL, Oracle, SQLite, SQL Server and generic
treat a backslash as an ordinary character, MySQL/MariaDB decode escapes
(keeping \% and \_), and ClickHouse/Snowflake decode escapes but keep unknown
ones. The tokenizer now follows dialect.StringEscape, and LiteralValue records
RawBackslash so the renderer writes a literal back under the same rules instead
of doubling every backslash on each format pass (non-idempotent, data
corrupting for regexes and paths). Unknown escapes no longer fail tokenizing.

Lexical coverage:
- operators ^ ^@ <=> <-> << >> == := and { }
- 0x hex numbers, leading-dot (.5) and trailing-dot (5.) numbers
- prefixed strings N'' X'' B'' E'' q'[]' and MySQL _charset'' (kept on the AST
  as LiteralValue.Prefix and written back)
- $ inside identifiers (Oracle/Snowflake/PostgreSQL), # in Oracle identifiers
- SQLite :name, ?NNN and [bracket] identifiers, ClickHouse {p:Type} parameters
- a leading UTF-8 byte order mark is skipped

The real-world corpus ratchet drops 64 known failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
FIRST, LAST, NEXT, FILTER, SHARE, SOURCE, TARGET, MATCHED, FINAL,
TEMPORARY and the role-option words were rejected where a column
reference starts (SELECT first, key FROM t), although no clause gives
them a keyword meaning there. Accept them as column names and qualified
name parts; KEY, INDEX and VIEW stay rejected in MySQL, MariaDB, SQL
Server and Oracle, where they are reserved, and none of the words
becomes a function name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…pping them

LIMIT/OFFSET/FETCH values are now expressions when they are not plain
integers ($1, ?, @n, -1, 1 + 1, subqueries), with LIMIT ALL, OFFSET before
LIMIT, OFFSET n ROWS, FETCH FIRST ROW ONLY and SQLite LIMIT a, b. The new
SelectStatement fields (LimitExpr, OffsetExpr, LimitAll, OffsetRows, ...)
sit beside the existing *int fields, which stay filled for integers.

The formatter and SelectStatement.SQL() now render every SELECT field:
QUALIFY, START WITH / CONNECT BY, PREWHERE, ARRAY JOIN, TOP inside derived
tables (keeping TOP (n) parentheses), OFFSET ... ROWS before FETCH, and
CROSS/OUTER APPLY without a trailing JOIN. A reflection test fails when a
new SelectStatement field is not rendered by both renderers.

New syntax, dialect-gated:
- MySQL select modifiers (SQL_CALC_FOUND_ROWS, STRAIGHT_JOIN, ...),
  STRAIGHT_JOIN joins, JOIN without ON, LOCK IN SHARE MODE, INTO
  @vars / OUTFILE / DUMPFILE; MariaDB/Oracle FOR UPDATE WAIT n
- SELECT ... INTO table (SQL Server, PostgreSQL) and WHERE / GROUP BY /
  HAVING without FROM
- GROUP BY ... WITH ROLLUP / CUBE / TOTALS kept as written (MySQL rejects
  ROLLUP(...)), GROUP BY ALL (Snowflake, ClickHouse, BigQuery)
- ClickHouse LIMIT n BY, LIMIT n WITH TIES, FORMAT, INTERPOLATE, GLOBAL and
  ANY/ALL/ASOF/SEMI/ANTI join strictness, * EXCEPT / REPLACE
- Snowflake TOP n, * EXCLUDE / REPLACE / RENAME / ILIKE, ASOF JOIN ...
  MATCH_CONDITION
- SQL Server FOR JSON / FOR XML / FOR BROWSE, OPTION (...) query hints,
  HASH/LOOP/MERGE/REMOTE join hints; Oracle CROSS/OUTER APPLY, ORDER
  SIBLINGS BY, START WITH after CONNECT BY; PostgreSQL ORDER BY ... USING op
- a comma-separated table after an explicit join

Non-integer LIMIT literals such as 10.5 are no longer truncated to an int;
they are kept verbatim in LimitExpr so formatting never changes the row
count. The real-world corpus ratchet drops 95 known failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
A bare GRANT, KILL or EXEC was accepted as a complete verbatim
statement. Require at least one argument token except for the
statements that are complete on their own (VACUUM, ANALYZE, REINDEX,
CLUSTER, CHECKPOINT, and T-SQL THROW, RETURN, REVERT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… calls and typed literals

Primary expressions and function-call forms that failed to parse or
silently changed meaning on format now round-trip:

- Keyword-named functions: LEFT/RIGHT, GROUPING, TRUNCATE, DEFAULT, ROW,
  INSERT, ClickHouse any/range, Snowflake HASH, MySQL INTERVAL(n, ...).
- Qualified function calls (pg_catalog.now(), auth.uid(), pkg.fn()) and
  the Oracle (+) outer-join marker (OuterJoinMarker).
- Typed literals DATE/TIME/TIMESTAMP '...' (any type name in PostgreSQL)
  as TypedLiteral; SQL Server N'...' strings (Type "nstring"); adjacent
  string literals concatenated in MySQL, MariaDB and PostgreSQL.
- SQL-standard special forms via KeywordArgument: SUBSTRING FROM/FOR,
  POSITION IN, TRIM BOTH/LEADING/TRAILING ... FROM, OVERLAY PLACING,
  CONVERT USING, TRY_PARSE/PARSE AS, COLUMN_GET AS, LISTAGG ON OVERFLOW.
- GROUP_CONCAT SEPARATOR is kept in FunctionCall.Separator instead of
  being rendered as an extra argument (which changed results).
- Oracle KEEP (DENSE_RANK FIRST|LAST ORDER BY ...) as FunctionCall.Keep;
  CONNECT_BY_ROOT and PRIOR in the select list.
- INTERVAL <expr> <unit> keeps the value unquoted (IntervalExpression
  ValueExpr/Unit), so MySQL INTERVAL 1 DAY no longer formats as the
  invalid INTERVAL '1 DAY'; legacy Value text is preserved.
- MATCH ... AGAINST is a MatchAgainstExpression (was rendered as
  AGAINST AGAINST(...)).
- NEXT VALUE FOR, MySQL @@[scope.]var and @v := expr.
- ROW(...), nested ARRAY[[...]], ARRAY(<set operation>), bare bracket
  array literals with subscripts (ClickHouse, Snowflake), ClickHouse
  tuple element access (t).1, CAST(x, 'Type') and extract(s, re).
- TRY_CAST is rendered as TRY_CAST, CAST target types accept array
  suffixes and multi-word types (DOUBLE PRECISION, SIGNED INTEGER).

The real-world corpus ratchet drops 123 known failures; three move to a
later stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…ble functions

Formatting for SQL Server or Oracle now turns LIMIT / OFFSET expression
values ($1, @n, :n) into TOP (expr) or OFFSET expr ROWS FETCH NEXT expr ROWS
ONLY, as it already did for integers, and drops LIMIT ALL. The right side of
CROSS APPLY / OUTER APPLY may be a table-valued function call such as
sys.dm_exec_sql_text(qs.sql_handle) in dialects whose FROM clause does not
otherwise take function calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…k new operator nodes

- COLLATE "en_US.utf8" was written back as en_US."utf8" (a schema-qualified
  name): CollateExpression now carries the qualifier in a Schema field, so a
  dot inside a quoted name stays part of the name.
- CREATE INDEX ... (a COLLATE x): CreateIndexStatement.SQL() dropped the
  collation, and the formatter wrote COLLATE "C" as bare C (PostgreSQL folds
  it to c). IndexColumn gains CollateQuoted; both writers emit the collation.
- transform's expression walker now descends into AT TIME ZONE, COLLATE,
  ANY/ALL/SOME over arrays, the ClickHouse conditional and LIKE ... ESCAPE, so
  QualifyColumns / ReplaceTable reach their operands (incl. = SOME (SELECT)).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
- ClickHouse: reject a second LIMIT ... BY, LIMIT n WITH TIES BY, and an
  OFFSET written before LIMIT n BY instead of silently dropping or moving
  a clause.
- Snowflake: ASOF JOIN ... MATCH_CONDITION (...) no longer requires ON.
- Render qualified EXCLUDE / EXCEPT names part by part so t.a does not
  become the single identifier "t.a".
- Release the pooled children of a StarExpression (pass-through case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… a name

CALL with no procedure fell back to a verbatim UtilityStatement, SET = 1
recorded "=" as the variable name, and SAVEPOINT / RELEASE [SAVEPOINT] /
ROLLBACK TO [SAVEPOINT] / SAVE TRAN without a name parsed as valid. They
now report errors like GRANT and KILL with no arguments.

isKeywordColumnName now matches the token type before peeking at the next
token, so literal operands no longer pay for a token copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
Review follow-ups for the primary-expression and special-form work:

- Accept the ClickHouse NANOSECOND unit in INTERVAL <expr> <unit>; the
  stricter unit check had started rejecting it.
- Release pooled children under IntervalExpression.ValueExpr, FunctionCall
  KEEP ORDER BY, ARRAY(<set operation>) and the new unpooled nodes
  (KeywordArgument, MatchAgainstExpression, OuterJoinMarker,
  ConnectByRootExpression, AssignmentExpression).
- Have the AST security scanner descend into the new nodes. MATCH ...
  AGAINST (SLEEP(5)) was reported while it parsed as a BinaryExpression and
  was missed once it became a MatchAgainstExpression.
- Match schema- and package-qualified calls (pg_catalog.pg_sleep,
  DBMS_LOCK.SLEEP, UTL_HTTP.REQUEST, sys.xp_cmdshell) in the scanner, now
  that qualified calls parse in expressions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
# Conflicts:
#	testdata/realworld/known_failures.txt
# Conflicts:
#	cmd/gosqlx/cmd/format_corpus_test.go
#	testdata/realworld/known_failures.txt
# Conflicts:
#	testdata/realworld/known_failures.txt
# Conflicts:
#	testdata/realworld/known_failures.txt
# Conflicts:
#	cmd/gosqlx/cmd/format_corpus_test.go
#	pkg/sql/ast/ast.go
#	pkg/sql/ast/sql.go
#	pkg/sql/parser/dml_update.go
#	pkg/sql/parser/expressions_literal.go
#	testdata/realworld/known_failures.txt
BenchmarkRealWorldParse / BenchmarkRealWorldParseFormat run every
parseable statement of testdata/realworld through ParseWithDialect (and
FormatAST) in its own dialect, giving a production-shaped throughput and
allocation baseline across all eight dialects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…tatements

SplitStatements split on every top-level ';', so trigger and procedure
bodies were cut at the first inner ';' and multi-batch T-SQL scripts using
GO merged into one statement.

The splitter now tracks BEGIN ... END nesting (plus CASE, IF/LOOP/WHILE/
REPEAT inside a block, Oracle DECLARE and IS/AS sections, PostgreSQL BEGIN
ATOMIC) and keeps the whole compound statement together. Transaction
statements (BEGIN; BEGIN TRANSACTION; BEGIN IMMEDIATE; ...) are not blocks,
and a script whose blocks do not balance falls back to plain semicolon
splitting. Client batch separators are honoured and consumed: a GO [n]
line for SQL Server, the DELIMITER directive for MySQL/MariaDB and a lone
"/" line for Oracle. Plain scripts split byte-for-byte as before, and the
splitter still allocates only its result slice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…eject truncated ones

SQL Server batches without semicolons ("SET NOCOUNT ON SELECT * FROM t")
swallowed every statement after a SET, USE, transaction, PRINT, EXEC,
DECLARE, THROW, RAISERROR, WAITFOR, DBCC or RETURN into its text, so the
linter, scanner and analyzer never saw them. Such a statement now ends at
a top-level word that starts a statement (SELECT, INSERT, UPDATE, DELETE,
MERGE, WITH CTE, CREATE, ALTER, DROP, SET, EXEC, IF, BEGIN, ...), outside
parentheses and not after FOR / AS (cursor declarations) or a "." Options
such as WITH MARK, WITH RECOMPILE and WITH NOWAIT still continue the
statement. Statements that own a body (IF, WHILE, BEGIN blocks, END) and
GRANT / REVOKE keep running to the semicolon. Other dialects are unchanged.

The verbatim fallback no longer accepts truncated input: SET with nothing
after the variable name (SET GLOBAL, SET @x), SHOW alone or with only a
modifier, and statements that end on LIKE, WHERE, FROM, IN, TO, a comma or
an assignment operator are now parse errors, so gosqlx validate reports them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…LACE pool release

- SQL Server MERGE ... OUTPUT ... INTO @t now uses parseDMLOutputClause;
  MergeStatement gains OutputInto, rendered by the AST writer and formatter.
- MySQL and MariaDB reject DELETE t WHERE ... (FROM is required) and
  DELETE FROM t1, t2 WHERE ... (several targets need USING); both used to
  be accepted and rewritten or passed through. Oracle and SQL Server keep
  FROM-less DELETE.
- PutMergeStatement releases a derived-table source or target subquery and
  the OUTPUT INTO columns. Merge and Replace release now follow the #556
  releaseDrawn policy: ReleaseAST puts only the node types the parser draws
  (the parser allocates MergeStatement with a literal), while the exported
  Put functions still return everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…ATE and DELETE

The transforms only touched UpdateStatement/DeleteStatement.TableName and
WHERE, so MySQL multi-table forms, UPDATE ... FROM and DELETE ... FROM /
USING produced SQL that still named the old table, with no error.

ReplaceTable now renames the table in further targets, JOINs, FROM / USING
lists and in column qualifiers of SET, OUTPUT, ORDER BY and RETURNING.
AddTableAlias applies the alias wherever the statement names the table,
points the qualifiers at the alias (a table with an alias cannot be named
any more) and renames DELETE t FROM ... targets to the alias. A table that
appears twice (self join) returns an error instead of taking one alias for
both instances.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…ormat

Follow-ups to the dialect string/escape work:

- Unconfigured tokenizer (no dialect) is strict again: \\ \' \" \` \n \r \t
  decode, any other backslash escape is an error (as before the escape
  rework), so gosqlx.Parse/Format and the CLI 'format' without --dialect
  refuse 'C:\dir' instead of rewriting it to a different PostgreSQL value.
  New zero-value mode dialect.StringEscapeStrict; unknown dialects map to it.
- NUL is never silently dropped: escaping literals write it as \0 (E-strings
  as \000). A raw NUL/Ctrl-Z inside a string, or an escape decoding to one,
  is rejected where the formatter cannot write it back (no-escape dialects,
  unconfigured, Snowflake/Redshift; Ctrl-Z in ClickHouse).
- Snowflake and Redshift get StringEscapeDropUnknown: the backslash before
  an unrecognised character is ignored (Snowflake docs: "the backslash is
  ignored"; Redshift's documented LIKE default escape is written '\\\\'),
  with complete octal/\xhh/\uhhhh escapes; dialect-dependent short forms
  are rejected.
- ClickHouse no longer decodes \Z (kept as written, like other unknown
  escapes) and decodes \a \v \f and ASCII \xHH.
- Triple-quoted strings are BigQuery-only, so '''a' tokenizes (value 'a)
  and '''abc''' keeps its quotes in every other dialect.
- Hex literals (0x1E) are typed int, not float.
- A delimited identifier containing '$' keeps its delimiters (standard SQL
  and BigQuery do not accept '$' in bare names); bare ones stay bare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
Transpile wrote each string literal with the SOURCE dialect's escaping and
introducers. A PostgreSQL 'x\' followed by more text closed early in MySQL
(the rest of the statement ran as code), 'C:\dir' lost its backslash, and
E'..', q'[..]' and _utf8mb4'..' reached targets that reject them.

After the rewrite rules, every string literal (found by a reflective walk,
so a node with an incomplete Children list cannot hide one) gets the
target's backslash mode, and introducers the target lacks are dropped:
E'..' outside PostgreSQL, q'[..]' outside Oracle, a charset introducer
outside the MySQL family. The decoded value is unchanged and re-escaped
for the target. A NUL or Ctrl-Z the target cannot spell (NUL where
backslashes do not escape; \0/\Z outside MySQL where they read
differently) is an error instead of being dropped or changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…atements

ALTER TABLE previously accepted one unqualified table name and one of
ADD COLUMN / ADD CONSTRAINT / DROP COLUMN / DROP CONSTRAINT / RENAME / ALTER,
with ALTER COLUMN reading its sub-action as a column definition (so DROP
DEFAULT formatted as a column of type "DROP").

- Qualified names, IF EXISTS, ONLY, ON CLUSTER (ClickHouse) and the MariaDB
  ONLINE / IGNORE modifiers; comma-separated action lists in
  AlterStatement.Operations (Operation stays the first action).
- ADD [COLUMN] [IF NOT EXISTS] with the optional COLUMN keyword, Oracle and
  MySQL parenthesized lists, SQL Server column lists, FIRST / AFTER; ADD
  [CONSTRAINT] PRIMARY KEY / UNIQUE / FOREIGN KEY / CHECK with CLUSTERED /
  NONCLUSTERED, NOT VALID and other trailing attributes, WITH [NO]CHECK;
  ADD / DROP INDEX / KEY (MySQL, MariaDB, ClickHouse).
- DROP COLUMN [IF EXISTS] lists, CASCADE / RESTRICT, DROP PRIMARY KEY /
  FOREIGN KEY / CHECK; MODIFY, CHANGE, RENAME COLUMN / CONSTRAINT / TO / AS.
- ALTER COLUMN SET / DROP DEFAULT with any expression, SET / DROP NOT NULL,
  [SET DATA] TYPE t [USING e] and other sub-actions; full definitions for
  SQL Server / Oracle.
- Dialect-gated long tail (ENABLE / DISABLE / VALIDATE, OWNER TO, SWAP WITH,
  CLUSTER BY, SET props, MySQL table options, partition operations,
  ClickHouse MODIFY TTL / SETTING and partition actions) kept as a keyword
  phrase plus text as written; ClickHouse UPDATE / DELETE mutations parsed.
- ALTER INDEX / SESSION / SYSTEM / USER / DATABASE / SCHEMA / VIEW /
  WAREHOUSE / TASK / ... are kept as written in a UtilityStatement.
- The formatter renders every action; pkg/schema applies multi-action and
  multi-column ALTERs and CHANGE renames.

Removes 130 statements from the real-world known-failures list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…difiers

Set operations:
- INTERSECT binds tighter than UNION/EXCEPT/MINUS (precedence climbing);
  SQLite and Oracle keep equal-precedence left-to-right grouping
- record explicit DISTINCT quantifiers and MINUS spelling (rendered as
  EXCEPT only for a target dialect without MINUS)
- VALUES operands; right-nested INTERSECT renders without parentheses so a
  trailing ORDER BY/LIMIT keeps applying to the whole query

CTEs: render MATERIALIZED / NOT MATERIALIZED and ClickHouse scalar CTEs in
the formatter; accept VALUES and parenthesized query bodies.

Table factors:
- render ClickHouse FINAL, Snowflake AT/BEFORE/CHANGES, FOR SYSTEM_TIME and
  MATCH_RECOGNIZE in the formatter (previously dropped)
- store TABLESAMPLE / SAMPLE (method, args, PERCENT/ROWS, REPEATABLE/SEED)
- PostgreSQL ONLY, Oracle @dblink and AS OF / VERSIONS BETWEEN flashback,
  MySQL PARTITION (...) and index hints, SQLite INDEXED BY / NOT INDEXED,
  SQL Server bare (NOLOCK) hints, INDEX(ix) hints and FOR SYSTEM_TIME
  (incl. CONTAINED IN)
- VALUES derived tables, column alias lists on any derived table or table
  (typed lists for PostgreSQL record functions), table functions for every
  named dialect, keyword-named function arguments (FLATTEN outer =>)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
Ajit Pratap Singh and others added 2 commits October 26, 2026 13:19
Remove the SQL Server check that rejects a statement running across a
line holding only GO [n] (and its invalid-repeat-count message), with its
tests. A GO line inside a statement reads as an identifier again, as on
main. The rule is a breaking change and is reviewed separately on top of
this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
sqlcmd ends the batch at any line holding only GO [n], so a statement
that runs across such a line is two batches to the server. The parser
used to read the line as an identifier (SELECT a,\nGO\nFROM t became the
column list a, go), which formats into SQL that sqlcmd splits elsewhere.
It now reports "GO batch separator inside a statement", and GO with an
invalid repeat count (GO 0, GO 1.5) alone on a line reports "invalid GO
repeat count". Recovery parsing resumes at the GO line.

BREAKING CHANGE: SQL Server input with a line holding only GO [n] inside a
statement no longer parses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@vercel

vercel Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
gosqlx-website Ignored Ignored Preview Oct 26, 2026 10:33pm UTC

@github-actions github-actions Bot added documentation Improvements or additions to documentation parser labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 24s —— View job


Review of PR #868

  • Read the diff and the internal/batch.Line helper it relies on

Verdict: looks good. I have a few non-blocking suggestions. I did not run the tests.

What I checked

  • Detection: batchLineError has a cheap prefilter (identifier token, length 2). It then defers to batch.Line. batch.Line rejects quoted identifiers ([go], "go"), GO sharing a line with other tokens, and tokens without source positions. This keeps false positives low, and the same-line cases are covered by tests.
  • Dialect gating: the check runs only under IsSQLServer(), both at the call sites and inside the helper. Other dialects are untouched, and the PostgreSQL case is tested.
  • Cost: each statement is scanned once over its own token range, so the total work is linear in the input.
  • Error path: when the statement parser failed, the scan includes currentPos. A GO 0 or GO 1.5 line therefore gives the "invalid GO repeat count" error instead of a misleading FROM or end-of-statement error. "alias position" (SELECT a\nGO 2\nFROM t) correctly reports a FROM error. GO ends the first batch there, so that's expected.
  • Recovery: p.seek(at) followed by continue resumes at the GO token. The next iteration then parses it as a BatchSeparator, and the recovery test asserts this.

Suggestions (non-blocking)

  1. Duplicated call-site logic: parser.go has the same start := p.currentPos … batchLineError block in parseTokens and parseContextTokens. A small helper such as p.parseStatementChecked() would keep the two from drifting. The // batchLineError returns err unchanged elsewhere comment also sits next to an IsSQLServer() guard that the helper already performs. One of the two checks is redundant.
  2. Progress guarantee in recovery: p.seek(at) relies on at > stmtStartPos. If at equalled stmtStartPos, the loop would not advance. That should not happen, because a statement starting at a GO line is parsed as a BatchSeparator first. A short comment or an assertion would make the invariant explicit.
  3. Test gaps:
    • Nested constructs spanning a GO line: BEGIN … GO … END, a CTE, and a subquery. sqlcmd splits those too, but it would be good to pin the behavior.
    • parseContextTokens: no test hits the context-aware path.
    • Recovery after a failed parse (err != nil) containing a GO line. Only the successful-parse branch is covered.
  4. Changelog: the breaking change is documented under "Behavior changes". Consider also linking it to fix(dialect): complete real-world SQL audit and preserve round trips #858 and fix(dialect): parse, format and transpile the real-world corpus in all 8 dialects #867 for traceability.

No security or performance concerns.

Ajit Pratap Singh and others added 13 commits October 27, 2026 03:03
… rune

Remove a comment that described a check this branch does not contain,
report malformed and duplicate known_failures.txt entries instead of
skipping them, note that statement indexes shift when a corpus file
changes, and truncate report text on rune boundaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… huge DELIMITER

Memoise failed Oracle q-quote searches per closing byte so unterminated
literals cost one scan in total, and cap the MySQL DELIMITER token at 32
bytes (longer lines are ordinary text).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…tatement starts

- Optimizer hints: occurrence signatures are computed once per statement and
  keyword and shared by every hint; source/rendered signatures are compared
  once per keyword (occMatcher); the statement is written with one
  strings.Builder. Placement semantics and ErrUnplaceableComment refusals are
  unchanged.
- locateComment finds the item by binary search instead of a linear scan.
- The parser records the source start of every top-level statement in
  ast.AST.StatementStarts (reset on release, preallocated with the pooled
  AST). CommentLayout.Starts (filled via formatter.StatementPositions at both
  callers) cuts the source at those starts, so procedure/trigger bodies with
  ';' and T-SQL statements without ';' keep their comments. Disagreement falls
  back to the documented hoist; nil Starts keeps the ';' split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… subtrees, honour release mode

- MERGE: write the derived-table column alias list after SourceAlias in
  ast and formatter renderers.
- Children: CreateTableStatement (ENGINE clauses and args, CLONE point,
  partition bound, options, PERIOD identifiers), CreateMaterializedView
  (engine clauses/options/column defs), SelectStatement (TOP, ARRAY JOIN).
  TableOption gains ValueArgs so parenthesized ENGINE arguments are
  walkable while Value keeps the rendered text.
- AST.SQL()/stmtSQL fall back to ToSQL() then TokenLiteral() instead of
  skipping ToSQL-only statements (sequences, unsupported).
- CREATE TABLE release helpers honour releaseDrawn (#556).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…t one

- Outside SQL Server, SET/USE/CALL/COPY/VACUUM/ANALYZE/LOCK/DO/GRANT/
  REVOKE and closed CREATE/ALTER/DROP kinds end at a statement-starting
  word on a new line (SET/USE also mid-line); the missing ';' is then
  an error, as after "SELECT 1\nSELECT 2", unless a transaction
  statement follows on its own line.
- Transaction statements are checked against a per-dialect grammar
  (PostgreSQL, MySQL/MariaDB, SQL Server, Oracle, SQLite, Snowflake,
  ClickHouse, BigQuery; generic = union) and end where it ends, so
  trailing words error and the next line's statement is parsed on its
  own. END/ABORT treat "generic" like no dialect.
- <=> as a comparison only in MySQL/MariaDB (PostgreSQL keeps pgvector's
  operator); == only in SQLite and ClickHouse.
- T-SQL: unclosed BEGIN blocks, an END that closes nothing, and IF/WHILE
  without a body or ELSE without a statement are errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
# Conflicts:
#	pkg/gosqlx/realworld_corpus_test.go
…t review fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 18s —— View job


Review of PR #868

Verdict: looks good. I have a few non-blocking notes. I read the diff but did not run the tests.

What I checked

  • Detection: batchLineError (pkg/sql/parser/statement_boundary.go) prefilters on identifier tokens of length 2. It then defers to batch.Line, which keeps false positives low. The scan covers each statement's own token range once, so total work is linear.
  • Dialect gating: the check runs only for SQL Server. Other dialects keep their old behavior.
  • Error path: when the statement parser failed, the scan includes currentPos. GO 0 and GO 1.5 therefore get "invalid GO repeat count" in place of a misleading FROM error.
  • Recovery: p.seek(at) and continue resume at the GO token, which is then parsed as a BatchSeparator.

Suggestions

  1. Recovery misses the error branch (recovery.go): the GO check runs only when err == nil. If a statement fails and the failure is caused by a GO line (for example SELECT a,\nGO\nFROM t where the parse stops later), recovery reports the generic parse error. It then uses the usual sync logic, which may skip past the GO line and swallow the next batch. parseTokens handles this case, so recovery and the strict path now differ. Consider calling batchLineError(stmtStartPos, err) in both branches and seeking to the GO token when it returns one.
  2. Dropped statement in recovery: when the GO error fires on a successfully parsed stmt, the code does continue without releasing or appending it. This is probably fine, since the AST nodes are garbage collected. If pooled nodes are expected to be released elsewhere, check that this path doesn't leak them.
  3. Progress guarantee: p.seek(at) requires at > stmtStartPos, or the loop would not advance. That holds because a statement beginning at a GO line parses as a BatchSeparator first. A short comment stating the invariant would help.
  4. Duplicated call sites: the startPos … batchLineError block appears twice, in parseTokens and parseContextTokens. The IsSQLServer() guard is also redundant, since the helper already checks it, and the inline comment next to it is confusing. A small helper would stop the two copies drifting apart.
  5. Test gaps:
    • There is no test of parseContextTokens.
    • There is no test of recovery after a failed parse that contains a GO line (see item 1).
    • There is no test of nested constructs such as BEGIN … GO … END, CTEs and subqueries.
  6. Changelog: consider linking fix(dialect): complete real-world SQL audit and preserve round trips #858 and fix(dialect): parse, format and transpile the real-world corpus in all 8 dialects #867 for traceability.

No security concerns. I don't expect a performance impact: it adds a linear scan, and only for SQL Server.

Ajit Pratap Singh and others added 4 commits October 27, 2026 03:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…ound

The 1 s bounds failed on race-instrumented macOS and Windows runners
although the splitter is linear. Time n and 4n and fail above 10x:
linear work grows ~4x, the old quadratic code measured 16-24x.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@ajitpratap0
ajitpratap0 changed the base branch from fix/dialect-audit-fixes to main October 26, 2026 22:32
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 17s —— View job


Review of PR #868

I read the diff but did not run the tests.

Verdict: the design is sound. I have a few non-blocking suggestions, and one of them is worth fixing before merge.

What I checked

  • Detection (statement_boundary.go): batchLineError prefilters on identifier tokens of length 2, then defers to batch.Line. Each statement's token range is scanned once, so the cost is linear.
  • Dialect gating: the check is SQL Server only, so other dialects are unchanged.
  • Error path: when the statement parser failed, the scan includes currentPos. GO 0 and GO 1.5 therefore get "invalid GO repeat count" in place of a misleading FROM error.

Suggestions

  1. Recovery ignores the error branch (recovery.go:210). The GO check runs only when err == nil. parseTokens calls batchLineError(startPos, err) for failed parses too. Take SELECT a,\nGO 0\nFROM t. Recovery reports the generic parse error and uses its usual sync logic, which can skip past the GO line and swallow the next batch. Strict mode and recovery therefore disagree. Call batchLineError(stmtStartPos, err) in both branches, and when it returns an index, report the error and p.seek(at).
  2. Progress invariant in recovery: p.seek(at) followed by continue only advances if at > stmtStartPos. That holds because a statement starting at a GO line parses as a BatchSeparator first. Add a one-line comment saying so, or guard with at > stmtStartPos, so a future change can't cause an infinite loop.
  3. Duplicated call sites (parser.go, in both parseTokens and parseContextTokens):
    • The startPos … batchLineError block is copied twice. A small helper such as p.parseStatementChecked() would stop the copies drifting apart.
    • The if p.IsSQLServer() guard is redundant, because the helper already checks the dialect. The _, err = discard also hides the index. Drop the guard and the confusing inline comment.
  4. Dropped statement in recovery: when the GO error fires after a successful parse, stmt is discarded without ast.ReleaseStatement or similar. This is a leak only if statements are pooled in this path. Check how the existing error branch handles it and match it.
  5. Test gaps:
    • No test covers parseContextTokens.
    • No test covers recovery after a failed parse that contains a GO line (see item 1).
    • No test covers nested constructs (BEGIN … GO … END, a CTE, or a subquery). sqlcmd splits those too, so a test would pin the behavior.
  6. Changelog: consider linking fix(dialect): complete real-world SQL audit and preserve round trips #858 and fix(dialect): parse, format and transpile the real-world corpus in all 8 dialects #867.

No security concerns. The performance cost is a linear scan on SQL Server only.

@ajitpratap0
ajitpratap0 merged commit 953138e into main Oct 26, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation parser

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant