diff --git a/.changeset/eql-json-tree-cascade-and-random-order.md b/.changeset/eql-json-tree-cascade-and-random-order.md new file mode 100644 index 000000000..c4894f2a1 --- /dev/null +++ b/.changeset/eql-json-tree-cascade-and-random-order.md @@ -0,0 +1,5 @@ +--- +"@cipherstash/eql": patch +--- + +EQL v4 `json_tree` (unreleased): a stored document's entries are now in a random order, root first, an object's members get a random permutation as their ordinals (the `o` the server probes a node's children at, which was a keyed rank), and its containment projection `q` is shuffled the same way, so the stored order no longer shows each row's tree, the size of every subtree or the byte order of sibling key names (#1357). A containment or equality needle is sent in a random order of its own too, so a query no longer shows sibling key order either. Every entry's ciphertext now also commits to its children's AEAD tags, so the client checks a subtree read's rows by the chain rather than by their positions, and a row swapped, dropped, added, reordered or moved beneath the node read is refused by the cipher. Nothing stored changes shape; `json_tree_children` returns a node's children in ordinal order (an array's elements by position), where it returned them by stored slot, and no other SQL function changes (three doc comments stop promising document order); documents sealed by the earlier build of this unreleased family no longer open. `->`, `->>` and an ordering on `eql_v4.ord_term(col -> sel)` at a wildcard path now return an unspecified element of the row (the same one every time), not element 0; read every element with `json_tree_path_query`. `->` on an object or array with members no longer opens on the client (the entry's ciphertext commits to its children, which `->` does not return): Stack Encrypt reports `JsonNotALeaf`, which is also what an altered entry looks like through `->`; read a container with a subtree read. `docs/reference/json-tree-leakage.md` says what the stored row still shows and what a read now proves. diff --git a/Cargo.lock b/Cargo.lock index 271420eac..390a49aad 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3404,9 +3404,11 @@ dependencies = [ "base85-simd", "chrono", "clww-ore", + "futures", "miette", "ore-rs", "proptest", + "rand 0.10.1", "rust_decimal", "serde", "serde_json", @@ -3419,6 +3421,7 @@ dependencies = [ "tokio", "trybuild", "uuid", + "vitaminc 0.8.0", "vitaminc-aead 0.8.0", "vitaminc-aead-frame", "vitaminc-encrypt 0.8.0", diff --git a/docs/fuzzing.md b/docs/fuzzing.md index ca130747b..86c212c3d 100644 --- a/docs/fuzzing.md +++ b/docs/fuzzing.md @@ -28,7 +28,7 @@ Current targets: | `fuzz:term-decode` | `stack-encrypt` | `term_decode` | the SEM term decoders (`EqualityTerm`, `MatchTerms`, `OreTerm`, `OpeTerm` `from_bytes`) | | `fuzz:check-record` | `stack-encrypt` | `check_record` | `dynamic::record::check_record` over an `Arbitrary`-derived plan and tree, against a model of the record rules (structure-aware) | | `fuzz:plan-build` | `stack-encrypt` | `plan_build` | `plan::FieldsBuilder::build` over `Arbitrary`-derived untrusted names (context, fields, identities); a built field's label must render and parse back, and passthrough names are never refused (structure-aware) | -| `fuzz:json-wire` | `stack-encrypt` | `json_wire_decode` | the JSON index's wire decoders (`JsonDocument`, `JsonNode`, `JsonProjection`, `JsonLeafQuery` from JSON text; `JsonHeader::from_bytes`), over base64 tags and base85 ciphertexts; an accepted document or node must round-trip through its wire form | +| `fuzz:json-wire` | `stack-encrypt` | `json_wire_decode` | the JSON index's wire decoders (`JsonDocument`, `JsonNode`, `JsonSelectorSet`, `JsonLeafQuery` from JSON text; `JsonHeader::from_bytes`), over base64 tags and base85 ciphertexts; an accepted document or node must round-trip through its wire form | Each target is a few lines — `libfuzzer-sys` hands a `&str` (or `&[u8]` for the byte decoders) to the parser via the `arbitrary` crate: diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index f8d869b20..c19160343 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -518,7 +518,7 @@ Two of the three items that gate a release are closed here; the third, the final 8. **The sealed root is a boolean option, on by default; its name is open.** Decided 2026-10-18, see "The sealed root" under the token chain. The two shapes are `d = 0` (chain from the root) and `d = 1` (document at the root, heads at the top-level fields); no other depth is exposed, because `d ≥ 2` leaks subtree sizes at rest and grows storage linearly. Working name "root node obfuscation", which says what it hides; alternatives that say what it stores are `sealed_root` or `root_document`. Phase 3 should measure the root copy against the three document shapes so the default is confirmed by numbers rather than by the estimate above. 9. **Native `@>` over the element-nested projection, built from tags or from a nested `sv`.** Both give identical leakage; the first keeps the flat `sv` every other function reads, the second drops two fields from the wire and changes `->` to a `$.**` lookup. Proposal: projection from tags first, measure both in Phase 2. 10. **Position-keyed parent tags on by default.** They remove sibling linkability at rest for one small integer per entry and `W × F` hashes per walk. Proposal: on, unless Phase 3 shows the walk cost matters on large documents. -11. **Key the label step of the token chain.** As written, a child's token is an unkeyed hash of its parent's token and its label, so whoever holds a token can test guesses of every key name beneath it (see "What a token is" under the token chain). Deriving the step under the index key, `PRF(json/token ‖ column, t(parent) ‖ 0x01 ‖ key)`, closes that at no cost to the server, which unwraps child tokens rather than deriving them. The same holds for anything else the client derives from a label under a token. Proposal: key it. +11. **Key the label step of the token chain.** As written, a child's token is an unkeyed hash of its parent's token and its label, so whoever holds a token can test guesses of every key name beneath it (see "What a token is" under the token chain). Deriving the step under the index key, `PRF(json/token ‖ column, t(parent) ‖ 0x01 ‖ key)`, closes that at no cost to the server, which unwraps child tokens rather than deriving them. The same holds for anything else the client derives from a label under a token. Proposal: key it. **Where this stands:** the label step is keyed through the member's selector (the 2026-10-24 change below), and a member's ordinal is a random permutation per document (the 2026-10-26 notes), so no keyed rank remains in the format. ## Design change (2026-10-24): derivable tokens, the self tag, and wildcard expansion on the client @@ -544,7 +544,7 @@ p(child) = H8("p" ‖ iv ‖ t(parent) ‖ LE32(o)) unchanged; o `H8` is the first eight bytes of SHA-256, as Phase 3d made the parent tag. Object keys are unique, so a member's selector alone separates it from its siblings, and the ordinal adds nothing to its token but the dependency on the other keys. Every element of an array shares one selector (the wildcard path's), so an element keeps its index. Since an element's ordinal already is its index, an element's token is unchanged. Chain heads and the root derive as today: under the sealed root a head's token is `PRF(json/token ‖ column, path)`, and with the root open the root's token is that PRF of `$` and a top-level field's follows the member rule from it. -**Ordinals stay, for enumeration only.** The parent tag `p` keeps the ordinal, and an object member's ordinal is still its keyed rank. Enumerating a node's children is still a probe of `o = 0, 1, 2, …` until the first miss, and the keyed rank still keeps a token's holder from ranking guessed key names against the ordinals the probe reveals. Only the token stops depending on it. +**Ordinals stay, for enumeration only.** The parent tag `p` keeps the ordinal, and an object member's ordinal is a uniformly random permutation of its object's members, drawn from the document's generator when it is sealed (`member_ordinals` in `packages/stack-encrypt/src/json/derive.rs`). Enumerating a node's children is still a probe of `o = 0, 1, 2, …` until the first miss, and the ordinals the probe reveals say nothing about the keys, in one row or across rows. Only the token stops depending on it. **The consequence is the point of the change: the client derives the token of any wildcard-free path from the path alone.** Head, then one hash per step, the step's selector from the path and, for a position, the index from the path. A token sent therefore always covers exactly the node asked for, and the gates have no reason to exist. @@ -699,3 +699,25 @@ The gated-walk column is the benchmark's in-server figure for the same scenario, - *The sealed root's children enumerate from nothing.* The top-level entries carry no parent tag, because the format predates the root having a token, so `json_tree_children` at `$` returns no rows and `json_tree_child_count` returns NULL on the default layout, and a developer counts top-level fields by reading `$` and opening it. Chaining the heads from the root's token, `p = H("p" ‖ iv ‖ t($) ‖ LE32(o))`, would make the root enumerate like every other node. **Decided against for now (owner, 2026-10-25):** a walk from the root is a capability over the whole document's shape that the format deliberately withholds, so adding it increases leakage; SteVec offered nothing like it either, and it waits for a customer to ask. - *Withheld rows are not detected.* Nothing in a row says how many nodes a read should find, so a database that drops rows makes a node look absent, or shortens a wildcard read. This is an integrity property against a lying database, outside the current threat model (confidentiality against an honest-but-curious one); closing it needs an authenticated count, for instance the array's child count read beside a wildcard. - *The entries × tokens `LIMIT` and the token cap bound a lying database's own work.* Forged member tags can make a walk visit one subtree repeatedly; the client refuses the repeated positions, so only the database's work is at stake, and with the default sealed root and budget the read stops at `budget + 1` rows regardless. The `LIMIT` is the only bound when the budget is 0 or the root is open, and the token cap (`TermError::JsonWildcardTooWide`) scales it: a one-level wildcard with bound N sends N + 1 tokens and two levels send N² + N + 1, so the cap is 8,192 (owner, 2026-10-25; it was 65,536, which no measurement approached, the largest read benchmarked sending 4,161): every one-level read up to 8,191 elements, two levels up to a bound of 90, the default of 64 included, and no third level. + +### Implementation notes (2026-10-26): the cascade, and a random stored order (#1357) + +The stored order was the last disclosure of the format that no option changed: entries depth-first, members in key byte order, so a row's sequence of selectors gave its tree exactly, and comparing rows gave the parent of every node, every subtree's size and the byte order of sibling key names. It was deferred because the client's integrity check leaned on it: a read's rows had to sit at consecutive positions below the node asked for, with the position bound into each AAD. The spike in #1371 showed the way out, and the report linked from #1357 the analysis; this is the change, made the format. + +1. **The cascade is the integrity rule.** Every entry's AAD (domain `stack-encrypt/json/entry/v2`) binds, beside the keyset, the column, the key tag, the selector and the slot `j`, the digest of its children's AEAD tags, `SHA-256("k" ‖ tag(child₀) ‖ tag(child₁) ‖ …)` in ordinal order, a tag being the last sixteen bytes of a ciphertext as vitaminc lays a leaf out. Sealing runs from the leaves up. A leaf, an empty container and the sealed root (whose members chain from nothing) open under the digest of no children. A node's tag covers its AAD, so it commits to every ciphertext beneath it. Opening a read places its rows the way the server found them, by the chain (the node a token names by its self tag, a node's children by its parent tags at ordinals 0, 1, 2, …, each under the member or element rule its self tag fits), opens each node under the digest of the children placed under it, and then checks it. Nothing reads a slot as a position, so the rows may arrive, and be stored, in any order. Nothing stored changes: the AAD is not stored and the tag is already in `c`. Code: `packages/stack-encrypt/src/json/cascade.rs` and `place` in `derive.rs`. + + What it proves: the rows beneath the node a read names are exactly the ones sealed there, complete, each under its parent and in its place. What it does not: the place of that node itself, which is checked by its selector and its decrypted label alone, since nothing above it is read. The three allowances of ADR 0010 stand (another row's entry; another element's node at the same selector at the top of a read; withheld rows), and ADR 0010 now says so for `json_tree`. Upward verification was weighed in the report and not taken: it needs the ancestors' tokens, which is the capability the token-list read removed. + + Which refusal comes from where. The cipher (`Error::Aead`): a row swapped, dropped, added, reordered or moved under another parent beneath the node read; a truncated or reordered whole document; another column. An entry `->` returned that does not open alone is `Error::JsonNotALeaf`: a container with children (its AAD covers tags `->` does not return) or an altered entry, which nothing outside the ciphertext tells apart, so the error names both and claims neither. The client (`Error::JsonWrongEntry`): rows of two documents and an entry returned twice, before any key is retrieved; a row no token's subtree reaches; a reported depth the chain does not give; a child whose self tag fits neither rule, or not the one its label names; a node at the top of a read whose selector or label is not its path's; the sealed root anywhere but at `$`; a container whose plaintext child count is not the number of children it opened over. The hostile-row tests (`tests/json_read_checks.rs`, the PostgreSQL oracle) assert each of these exactly, and find rows by their paths rather than their slots. + +2. **A random stored order is the format.** Not an option, a flag or a switch. The root stays at slot 0, where the sealed-root fallback (`sv -> 0`) finds it; every other entry is placed by a uniformly random permutation of the slots 1…n, and `q`'s pair lists and element groups are shuffled the same way. The permutation is drawn per document at sealing time and thrown away: it is derived from nothing, keyed or not, because nobody needs to recompute it, and a permutation derived from anything stored (the `iv`) would be computable by the database. The generator is vitaminc's `SafeRand::from_entropy` with `next_below` (Lemire's reduction, bias at most n / 2⁶⁴) driving Fisher–Yates (`packages/stack-encrypt/src/json/order.rs`); it is not part of the format, since nothing stored depends on it. For fixtures, `StackCipherBuilder::json_order_seed` (behind `test-support`) draws every document's order from one stream started at a seed, through `SafeRand::from_controlled_seed`, and the deterministic test cipher sets it: `tests/fixtures/json_tree_vectors.json` now pins, per root option, the entries in their stored order, the path stored at each slot, the selector multiset, `q` as sets, and the tokens. The tags and tokens did not move. + + What stays visible at rest, and no order hides: the multiset of selectors (paths, node counts, array lengths, whether two rows have the same paths in the same numbers), `q`'s grouping by array element, which entries are heads under the sealed root (the head count, no longer the size of each head's subtree), each ciphertext's length, and the ordering terms at ordered paths. The leakage page drops its stored-order row and says this; the executable profile in `tests/json_leakage` models the new order, reading every world in an order an observer can rebuild without the stored one (`canonical.rs`). A containment or equality needle is shuffled the same way, with a fresh permutation per query drawn from the same `OrderSource` (`JsonQuery::of_terms` and `at_path`; the seed travels in the query's `JsonTerms`), so the query-time profile matches the at-rest one: in the walk's order a needle would show which of two sibling keys sorts first. `jsonb @>` is set containment over arrays, so the order changes no answer, and `JsonSelectorSet` (and so `JsonQuery`) compares as sets. Added at the owner's request after review, in the same PR. + +3. **First match at a wildcard path is unspecified (owner, 2026-10-26).** `json_tree_path_query_first`, and so `->`, `->>` and an ordering on `eql_v4.ord_term(col -> sel)`, returns the first entry in `sv` carrying a selector. At a wildcard path (every element under `$.a[*]`, at any depth) that is now an unspecified element, stable for a stored row but not element 0. Ordering or comparing over the elements of a wildcard path is written over all the matches (`json_tree_path_query`, `EXISTS` or `ORDER BY` over their ordering terms), and element order within a document is recovered after decryption from the labels, as it already was. No SQL changes. Two variants were considered and dropped by the owner: a fix-up after the shuffle that kept each wildcard selector's document-first entry first (it would have revealed, per wildcard selector, which entry is the document-first one), and writing the heads' digest into the sealed root's plaintext so that a client reading the root with its heads could verify them. The sealed root commits to nothing beneath it. + +4. **The keyset is loaded on a root-open decrypt.** Opening a whole document whose root is open walks the chain from the root's token, which needs the index key of the keyset it was sealed under. The open asks for it beside the data key, as a kind of request a `Pending` carries (`Request::load_keyset`), answered from the cipher's keyset cache or loaded from ZeroKMS on a miss through the path `StackCipher::keyset` uses; the loads run beside the data-key retrieval, and a load that fails fails only the documents under that keyset (`Error::KeysetLoad`). Settling a pending asks for a `DataKeySource` that is also an `IndexKeySource`, which every `StackCipher` is built over. The sealed root (the default) loads nothing. + +5. **An object's member ordinals are a random permutation.** The ordinal `o` in a member's parent tag, `H8("p" ‖ iv ‖ t(parent) ‖ LE32(o))`, which the server probes from zero to enumerate a node's children, was the member's keyed rank among its siblings (decision 11). Nothing reads a rank back: placement probes the parent tags and reads what is there, and a member's token does not depend on its ordinal. So each object's members now get a uniformly random permutation of `0..n` as their ordinals, drawn from the document's generator beside its stored order (`member_ordinals`, `shuffle::ordinals`). A token's holder, who sees the ordinals it probes, learns nothing from them about the keys, and cannot link a key to an ordinal across rows, which a rank under one token allowed. `RANK_DOMAIN` and `rank` are gone; the pinned vectors move in `order` and in each entry's `p` alone, the tags and tokens are unchanged, and the leakage profile's simulator draws per-row member orders (`member_order_shared_across_rows`). + +The SQL doc comments that promised "document order" (`json_tree_path_query`, `json_tree_children`, `json_tree_subtree`) now say what each returns: the stored order, ordinal order, and no promised order; no function changed. The benchmarks were not re-run: nothing stored and no SQL function changed. The client's added work is one SHA-256 per container over its children's tags, at sealing and at opening, and at opening the parent-tag probes the server already runs, which replace the position checks. diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index d950adc6c..dd54eb2c0 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -115,7 +115,7 @@ The "before" column is the profile at `9a7ca372c`; the "after" column is the por | `selector_precedence` | kept | which of two selectors comes first across rows: sibling key order | | `depth_recovery` | kept | an attack that reconstructs every row's depth sequence from the stored order alone | | `token_reach` | kept | per read, row and token: the node it finds and everything its walk links, with parents and depths | -| `ordinal_patterns` | kept | elements' ordinals in position order; members ranked consistently under one token | +| `ordinal_patterns` | replaced by `member_order_shared_across_rows` | whether one token's objects give their members one order of ordinals across rows, reported only below a `2⁻⁴⁰` chance bound: members' ordinals are now a random permutation per row | | `member_ordinals_follow_key_order` | kept | whether members' ordinals follow key order, reported only below a `2⁻⁴⁰` chance bound | | `token_derivability` | kept | which reads' tokens the database can derive from which others' (now by the member rule and by the element rule at an index below 4, from the tokens that find a node in some row) | | `transcripts` | kept | every query term with its tags named: determinism, shared selectors, needles; a read is now its token list and budget instead of a token and gates | diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 34e8e7007..21fc3b08d 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -2164,8 +2164,10 @@ dependencies = [ "base64ct", "base85-simd", "clww-ore", + "futures", "miette", "ore-rs", + "rand 0.10.2", "serde", "serde_json", "serdect", @@ -2174,6 +2176,7 @@ dependencies = [ "stack-kms", "thiserror 1.0.69", "uuid", + "vitaminc 0.8.0", "vitaminc-aead 0.8.0", "vitaminc-aead-frame", "vitaminc-encrypt 0.8.0", diff --git a/languages/golang/encrypt/guest/src/ops.rs b/languages/golang/encrypt/guest/src/ops.rs index aaba4a393..50d273fc1 100644 --- a/languages/golang/encrypt/guest/src/ops.rs +++ b/languages/golang/encrypt/guest/src/ops.rs @@ -51,7 +51,7 @@ use stack_encrypt::sem::MatchOptions; use stack_encrypt::target::IndexSpec; use stack_encrypt::ErrorPayload; use stack_encrypt::{CipherText, KeysetCipher, Label, Node, SealedValue, StackCipherText}; -use stack_kms::DataKeySource; +use stack_kms::{DataKeySource, IndexKeySource}; use zeroize::Zeroizing; use stack_guest_abi::last_error; @@ -101,7 +101,7 @@ pub async fn term( kind: u32, ) -> Result, u32> where - K: DataKeySource + Sync + 'static, + K: DataKeySource + IndexKeySource + Sync + 'static, { // The same proof every stack-encrypt leaf demands: an empty context is // `STATUS_ENCODING` here, before any derivation. @@ -178,7 +178,7 @@ pub async fn encrypt_record( plan: &[u8], ) -> Result, u32> where - K: DataKeySource + Sync + 'static, + K: DataKeySource + IndexKeySource + Sync + 'static, { let plan = parse_plan(plan)?; let tree = @@ -213,7 +213,7 @@ pub async fn decrypt_record( expected: Option