Repository navigation
feat(array): SQL relations, bulk load, region aggregates, narrow types and SRIDs; fix SQL output, redaction, audit and Calvin refusals - #422
Merged
Conversation
…n names Document psql superuser authentication, the 0.6 release status, and the engines that carry bitemporal columns. Clarify that AS OF SYSTEM TIME NULL works on ARRAY_SLICE but not ARRAY_AGG. Rename the stale NDARRAY_* references in array comments and the ARRAY_AGG planner error message to the ARRAY_* function names.
BitReader::read_bits now takes whole byte runs instead of one bit per step, rejects fields wider than 64 bits as corrupt, and leaves the cursor untouched when a read runs past the buffer.
Value::Geometry now carries an SRID alongside its shape. GeoJSON reads and writes a `crs` member for any SRID other than 4326, and the msgpack form gains an SRID field. nodedb-spatial gains a projection module (transverse Mercator and oblique Mercator families) and planar measures for projected SRIDs. WKB/EWKB handling moves into a module that preserves the SRID. R-tree bounding boxes are computed in the geometry's own space. SQL geometry functions read the SRID of each operand. Operands in different SRIDs, an unsupported SRID, or a coordinate outside a projection's domain raise a data exception (SQLSTATE 22000) rather than transforming implicitly; ST_Transform converts explicitly. A spatial index fixes its SRID at its first entry and refuses geometries in another SRID.
ARRAY_SLICE and ARRAY_PROJECT now return a typed relation, one row per cell, with the dimensions then the attributes as columns. WHERE, computed columns, GROUP BY, aggregates, ORDER BY, window functions and CTEs apply to those rows. Slice bounds push down into tile selection, and tile reads skip tiles by their bounding boxes. CTE inlining moves from the control-plane converter into the SQL planner. Arrays bulk-load through COPY ... FROM STDIN (text and CSV, optional header) and through a new ArrayPutBatch native opcode exposed as NodeDb::array_put_cells. Rows are checked as they arrive and written in bounded chunks, so memory stays flat regardless of input size. Other engines refuse bulk load with a typed error where it does not apply. Surrogate reservation for batched array writes is vectorised, array WAL records and the catalog are reworked for batch appends, and auto flush keeps freshly written cells visible to reads. Adds ordered-map msgpack encoding so rows have a stable column order.
…e ring refuses a chunk A refused partial chunk was dropped, so the collector could render a short result set as success. The stream now ends in a terminal ResourcesExhausted frame, and the collector discards the rows it holds.
…across shards and cores Add region-restricted array aggregation and projection-aware elementwise operations with a shared region and tile-key model in the planner, physical plan, and data-plane executors. Route them through the distributed array coordinator with new wire opcodes, group-key merging, and maintenance fan-out. Reads on the exchange path now gather per-core outcomes explicitly and apply vector prefilters from array reads. Payload merging returns an error on malformed msgpack instead of silently truncating, and graph neighbor hops that exceed max_results are refused rather than answered as a partial frame that a gather would wait on until its deadline.
Array attributes now store each cell at the declared width: UINT8, INT16, UINT16, INT32, UINT32, and FLOAT32 join INT64 and FLOAT64, with the usual SQL aliases. A value outside the width is refused with SQLSTATE 22003 on INSERT, COPY, and the native bulk load rather than wrapped or clamped. Columns encode by value range, with f32 columns using the 32-bit Gorilla codec. Region aggregates sum integer attributes exactly and refuse overflow, elementwise add/sub/mul stay exact on integers, and result columns report the matching pgwire type. Sync apply, cluster merge, and shard dispatch carry the new cell types.
The lexer emits NULL as its own token, so expecting it as a keyword rejected every NOT NULL attribute. Expect the token instead.
An unknown database or a catalog read error at startup was a non-fatal error, which left the connection open. On the SCRAM path ReadyForQuery is already sent, so the client read the error as the reply to its first statement. Send every startup refusal as FATAL so the connection closes. The TLS startup test now names the built-in database.
Without -d, psql asks for a database named after the user, and the server refuses the connection with 3D000.
…sion reads Aggregating timeseries scans now carry their SELECT-list output columns, so group rows are renamed to the client-visible aliases after HAVING, sort, and limit. The time key comes from the catalog. GROUP BY key aliases on the generic aggregate are applied the same way. Computed SELECT items over access paths that evaluate no expression (spatial scans, secondary-index fetches, search hits) are evaluated by a subquery tail that projects the final list. Sparse vector search carries the residual WHERE: the Data Plane widens the candidate window and the Control Plane keeps the top_k hits that pass. A read of a redacted column outside a plain projection (expression, predicate, sort, group key) is refused with 42501 on every engine. The plan collection helpers move into a plan_util directory.
A participant whose stage refuses its slice now votes abort with the typed refusal, encoded as opaque bytes the sequencer carries through the vote and the verdict. The coordinator decodes the refusal and the client sees the SQLSTATE a direct write renders, not a bare transaction rollback. When participants abort with different causes, a refusal outranks an unrefused participant error, and equal ranks resolve to the lowest vShard so every replica picks the same cause.
…itted writes A scheduler that starts must not stage before every write the gate admitted for its vShard reached its core. Such a write holds no key on the new lock table: none existed, or its keys were on a retired one. A transaction that staged first read the state before the write, and its redo then overwrote it. Every admitted write now holds a ticket that ends once the write reached its core or landed in its data group's log at a known index. A scheduler takes a start fence after its lock table registers and keeps its stage gate closed until the earlier tickets ended and their indexes applied. The propose tracker also releases an entry's held keys before it answers the proposer, so a proposer's next write never finds its own keys held.
An array aggregate now announces its columns: group, typed as the group dim, then result, typed by the reducer. A temporal aggregate also announces the truncated_before_horizon flag of its summary row. A timestamp group dim wrote raw milliseconds as an integer. It now writes the naive instant a slice row writes for the same dim, on both the Data Plane and the cluster finalize path.
…ells The coordinator partitioned a write into buckets of separately encoded cells, and each shard decoded and re-encoded the bucket into the flat array the Data Plane reads. A bucket is now written once as that flat msgpack array, and the shard passes it through. Cells of one tile share a tile key. A TilePrefixMemo keeps the last tile's indices and key, so planning, routing, COPY and memtable writes encode the space-filling curve once per run of cells in a tile. Bind keys for a put share one buffer.
…d columns A grouped SELECT list can now read a GROUP BY key expression and compute over aggregates and keys. A column that is neither a key nor inside an aggregate is refused with SQLSTATE 42803 (grouping_error) instead of reading a value the finalized group row does not carry. Computed select items take a numeric wire type when their operands are integer or float: arithmetic, math functions, negation, and CASE with branches of one type.
Parse planned a statement only to infer a result schema and swallowed any planning error, so a driver's prepare succeeded and the error surfaced at Execute. Parse now reports a parse-analysis error with its SQLSTATE, as PostgreSQL does: an unknown table, column, or function, an ambiguous column, and a column outside GROUP BY. An error a bound value can change, and one PostgreSQL raises while planning or executing, still surfaces at Execute. Parse strips a per-query ON DENY clause before planning, as Execute does.
…ex surrogate scopes Audit log - Entries are recorded into a buffer with backpressure, hash-chained, and persisted through the WAL before the catalog flush. A dedicated flush loop runs on a period and on early-flush requests, and flushes once more on shutdown. - Readers merge flushed and buffered entries in sequence order. - A position index over DML audit rows lets the event consumer resume without a full scan, and a hold keeps unconsumed events replayable. - Recorded entries export to the configured sink after the audit lock releases. Surrogate map - Bindings are keyed by a per-collection scope id in single byte-string keys, so a compare is one memcmp and a collection's rows are one key range. - The routed exchange binds each home answer in the asking node's catalog, and replicated binds carry the collection with each key's candidate surrogate.
…up column ARRAY_AGG now resolves its output columns fallibly. SUM, MEAN, MIN, and MAX accept integer and float attributes only; a STRING or BYTES attribute is refused with a type mismatch error. The group column takes the declared type of the group dimension.
HTTP, native, sync, and OTLP bearer authentication now share a single gate that audits each refusal and each login with the user, session, and peer address. Audit entries can carry the authenticated user and session. The system audit_log table prints the same snake_case event names as SHOW AUDIT, and SHOW AUDIT IN DATABASE applies its event_type filter.
Record calls used to append and sync the audit WAL under the audit lock, so every recorder paid its own fsync serially. Recording now stages entries (seq, hash chain, WAL append) under the lock, waits for the sync with the lock released, and then publishes the entries in seq order. A leader/follower group commit lets concurrent recorders share one sync, and a batch of entries needs a single sync. Readers, flushes and export see published entries only, so none observe an entry that is not yet durable. An entry whose append or sync failed is still published to keep the hash chain whole, and its record call returns the error.
An attr column now decodes against the row count of its tile and is refused as corrupt when its stored count differs, before any allocation. The validity bitmap is borrowed from the payload instead of expanded into a per-row vector, so a hostile count allocates nothing.
Check the stored count against the shared capacity helper before the output vector is allocated.
The approved capacity helper lives in wkb/read.rs after the module split.
farhan-syah
changed the title
farhan-syah
added
the
run-ci
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the Array engine usable end to end for raster and gridded data: bulk load, query as a SQL relation, aggregate over a region, and project between CRSs. Also fixes the defects a full demo rehearsal found across SQL output, redaction, audit, pgwire, and the Calvin write path.
Array and spatial
ARRAY_SLICEreturns a typed relation (dims and attrs as columns), so WHERE, expressions, GROUP BY, ORDER BY, LIMIT, joins and CTEs apply.ARRAY_PROJECTplans as a whole-domain slice.ARRAY_AGGtakes a box and a polygon zone;grouptakes its dim's type; non-numeric reducers are refused at plan time.ARRAY_ELEMENTWISE,ARRAY_FLUSH,ARRAY_COMPACTand the vector prefilter run across every shard and core. Array primaries spread over all cores.COPY <array> FROM STDINand nativeArrayPutBatch. Bulk load costs about 35.6k instructions per cell (was 112k).UINT8,INT16,UINT16,INT32,UINT32,FLOAT32. A numeric NULL stays NULL through flush.ST_Transformcovers 4326, 32647, 32648, 3375, 3376. Projected geometries measure in planar metres. Mixed SRIDs are refused (22000).SQL and pgwire
ROUND(AVG(x), 1)).psql ... -d default.Security and audit
plan_util/.VERIFY AUDIT CHAINchecks the whole chain.auth_successorauth_failure; bearer refusals do not feed lockout.SHOW AUDITprints snake_case event names and filters byevent_type.Write path
Breaking changes (pre-1.0, no migration)
surrogate_pk_v4, scope table), audit catalog rows. Data directories frommaindo not open cleanly.ArrayOp,ClusterArrayOp,VectorOp::SparseSearch,TimeseriesOp::Scan,QueryOp::Aggregate, and Calvin sequencer entries gained fields.SHOW AUDITevent names are snake_case.nodedb-liteneeds follow-up changes before it can move to these crates.Known limits
INSERTstill shows whether a redacted key exists.Test plan
cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features: 22431 passedcargo nextest run -p nodedb-cluster-tests --all-features: 511 passedcargo clippy --workspace --all-targets --all-features -- -D warnings, plus the threenodedb-clientfeature setscargo fmt --all -- --check,cargo deny checkscripts/ci/gates: Calvin determinism, format expects, plane separation, warm storage, Raft transport boundarywasm32check of the codec, columnar and strict crates; fuzz crate builds with its lock