From a5a886a41af16ae688d118af1b0ceb82d3603216 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 20:57:54 +1100 Subject: [PATCH 1/4] docs: capture Stack Encrypt API review --- .../2026-10-24-stack-encrypt-api-review.md | 684 +++++++++++++++++ ...26-10-24-stack-encrypt-context-examples.md | 723 ++++++++++++++++++ ...026-10-24-stack-encrypt-public-examples.md | 572 ++++++++++++++ 3 files changed, 1979 insertions(+) create mode 100644 docs/plans/2026-10-24-stack-encrypt-api-review.md create mode 100644 docs/plans/2026-10-24-stack-encrypt-context-examples.md create mode 100644 docs/plans/2026-10-24-stack-encrypt-public-examples.md diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md new file mode 100644 index 000000000..ad6afc78e --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -0,0 +1,684 @@ +# Stack Encrypt API review + +Started: 2026-10-24. Scope: the current Rust `stack-encrypt` API, starting +with encrypting one `i32` into opaque ciphertext and storing the result. + +Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md), +an API design draft covering encryption, persistence, decryption, and queries. +The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) +explore context sources and storage using Serde-inspired attribute syntax. + +This is a running issue log. Add a numbered section as each issue is found; +keep observations, proposed changes, and unresolved decisions distinct. +All issues below remain unimplemented. Individual decisions are recorded +within each issue; proposed APIs are illustrative, not implemented. + +## 1. Cipher trait mismatch and explicit context ergonomics + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` is an inherent method, not a trait method: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let ciphertext = keyset.encrypt(42_i32, "users/age").await?; +``` + +Vitamin C (`vitaminc-aead` 0.8.0) provides both `Cipher` and `AsyncCipher`. +Stack implements the synchronous `Cipher` trait on internal adapters after +fetching keys: `MintedKeys` seals leaves and `BoundKeys` opens them. Neither +public cipher implements `AsyncCipher`, so callers cannot use +`KeysetCipher` directly with generic Vitamin C code requiring that trait. + +Each internal adapter implements only half of `Cipher`: `MintedKeys::open` +and `BoundKeys::seal` always return errors. + +### Why the mismatch exists + +`AsyncCipher` already supports batching through `seal_many` and `open_many`; +batching alone does not explain the separate API. Its methods receive each +leaf's derived AAD, while ZeroKMS needs the original structured context's +descriptor. That original context is not carried by the derived leaf AAD. +Stack's pending-operation layer also combines key requests across documents, +fields, and rows, beyond the trait's per-document batch interface. + +These constraints need to survive any trait integration. The current +separation is deliberate, but leaves an interoperability gap and internal +implementations with unsupported operations. + +### Requested API direction + +Requiring an empty tuple for the common case of no application AAD is +unnecessary ceremony. Use distinct methods, following the convention used +elsewhere: + +```rust +keyset.encrypt(42_i32).await?; +keyset.encrypt_with_context(42_i32, "users/age").await?; +``` + +The no-context method should select the same empty context currently +represented by `()`. This distinction should exist even if `KeysetCipher` +continues to use inherent methods rather than implementing a Vitamin C trait. + +### Open design questions + +- Can a context-bound adapter implement `AsyncCipher`, or should Vitamin C's + interface preserve the original context explicitly? +- How does that integration preserve batching across multiple documents? +- Should sealing and opening be separate trait capabilities, avoiding methods + that are present but always fail? +- Should decryption expose the corresponding `decrypt` / + `decrypt_with_context` distinction? + +**Source:** [cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) +(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's batching +discussion). Vitamin C trait definitions are in `vitaminc-aead` 0.8.0, +`src/seal.rs`. + +## 2. The Node / SealedValue boundary complicates opaque ciphertext storage + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` returns `StackCipherText`, an alias for Vitamin C's +`Node`. Encrypting an `i32` produces `Single(SealedValue)`, but +the return type also admits sequences, maps, empty containers, and passthrough +nodes. It does not retain the plaintext type as a type parameter. + +| Capability | `StackCipherText` tree | `SealedValue` leaf | +| --- | --- | --- | +| Debug printing (`{:?}`) | Yes | Yes | +| Serde serialization/deserialization | No | Yes | +| Canonical byte encoding | None | `to_bytes()` / `from_bytes()` | +| Direct Base64 conversion | None | None; encode the canonical bytes separately | + +The leaf's canonical bytes include its keyset id, key retrieval metadata, +and local ciphertext. That versioned byte format is the storage commitment; +the leaf's Serde representation is not the same compatibility commitment. + +### Caller impact + +Even the scalar storage path requires exposing the container representation: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let StackCipherText::Single(leaf) = ciphertext else { + unreachable!("encrypting an i32 produces a single leaf"); +}; +let bytes = leaf.to_bytes(); + +// After loading the bytes: +let leaf = SealedValue::from_bytes(&bytes)?; +let ciphertext = StackCipherText::Single(leaf); +let value: i32 = cipher.decrypt(ciphertext, ()).await?; +``` + +The caller must handle a shape mismatch that cannot occur for this input, +extract the leaf, and reconstruct the node to decrypt. There is no direct +`ciphertext.to_bytes()` or `serde_json::to_string(&ciphertext)` path. Base64 +adds a separate encoding step. Collections additionally require the caller +to choose a representation preserving the tree structure and node variants. + +Stored bytes are not sufficient by themselves to decrypt: the caller still +needs authorized ZeroKMS access and the same application context. The storage +API should make that requirement clear without exposing tree internals for +the simple scalar case. + +### Proposed direction and open decisions + +Provide a straightforward opaque ciphertext storage round trip for scalar +values, with direct byte conversion and reconstruction. Decide whether this +means a dedicated scalar result type, a wrapper, or conversion helpers on +the existing API. + +Also decide whether Base64 and Serde should be supported conveniences, and +whether whole-tree serialization belongs in Stack or remains caller-owned. +If it remains caller-owned, make that distinction explicit and keep it out +of the simple scalar path. Preserve the existing canonical leaf encoding; +an ergonomic improvement should not require changing stored ciphertext. + +**Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) +(`StackCipherText`, `SealedValue`) and +[storage documentation and public re-exports](../../packages/stack-encrypt/src/lib.rs). +Vitamin C's container definition is in `vitaminc-aead` 0.8.0, +`src/container.rs`. + +## 3. Plaintext-side encryption has different syntax and context requirements + +**Status:** Open. + +### Current behavior + +The current Vitamin C `Encrypt` trait describes a plaintext through `visit`; +it does not provide an `encrypt` method. Importing Stack's re-export of that +trait does not enable `10_i32.encrypt(&keyset)`. Vitamin C 0.8.0 documents +this trait as replacing the older visitor-based trait that drove a cipher +directly. Its `ContextTag` wrapper has encryption methods, but those require +Vitamin C cipher traits and do not make the syntax available on bare integers +or accept Stack's public ciphers directly. + +Stack supplies its own blanket `EncryptInto` extension trait: + +```rust +use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; + +let ciphertext: StackCipherText = 10_i32 + .encrypt_into_with_context(&keyset, nonempty!("users/age")) + .await?; +``` + +The output type selects the `EncryptFrom` implementation. For +`StackCipherText`, that implementation requires an `Encrypt + Clone` source +and an `AeadContext`, which must be nonempty. + +Although `EncryptInto` also has `encrypt_into(&keyset)`, that method requires +the target's context to implement `Default`. `AeadContext` does not, so this +no-context form cannot produce `StackCipherText`. It is available for other +targets with default contexts, such as declarations carrying their own +contexts. + +The scalar derive's implicit context choice is also a discoverability concern: +`#[stash(plaintext = i32)]` generates `type Context = CallerContext`, even +for a record containing only a `StackCipherText` field (whose own context is +`AeadContext`). The generated operation converts between those types. The +reviewer found that hidden choice surprising; whether it needs an API change +or clearer diagnostics/documentation remains undecided. The `Default` bound +belongs to `EncryptInto::encrypt_into`, not to the `EncryptFrom` trait itself. + +### Caller impact and open decisions + +Plaintext-side encryption exists, but is not a symmetric spelling of +`keyset.encrypt(value, context)`: it uses different method names, selects an +output through a target trait, adds a `Clone` bound for this output, and +refuses the empty context accepted by the inherent method. + +Resolve the differing empty-context requirements alongside issue 1; +renaming alone would not make the two paths equivalent. This inconsistency +is separate from the receiver choice below. + +### Receiver choice: plaintext or keyset? + +**Decision (2026-10-24):** use only the keyset receiver for the ordinary +`encrypt` method: + +```rust +tenant_keyset.encrypt(value).await?; +``` + +Do not add a plaintext-side `value.encrypt(&keyset)` counterpart. +**`EncryptInto` remains.** This decision concerns only the `encrypt` method, +not removal of the target-directed encryption API. Implementation is pending; +the current inherent method still requires its context argument (issue 1). + +The two receiver forms express the same inputs and operation. Neither inherently provides +better type safety, ownership behavior, performance, or batching; those +depend on the signatures and implementation, not which argument is `self`. + +| Consideration | Plaintext receiver | Keyset receiver | +| --- | --- | --- | +| Mental model | Transform this value using a capability | Ask this encryption capability to process a value | +| Discoverability | Requires importing an extension trait; methods appear on plaintext types | Encryption methods are found on the encryption handle | +| Fluent composition | Reads naturally after an expression producing plaintext | Reads naturally when selecting a keyset before an operation | +| Operational scope | Keyset and its authority appear as an argument | Keyset and its authority lead the call | +| API surface | Adds methods to primitives and application types | Keeps execution methods on library-owned handles | +| Generic integration | Can work through a cipher trait | Can work through a cipher trait | + +**Rationale:** Stack performs an asynchronous operation through a configured KMS +capability; putting that capability first makes its scope visible and gives +callers one place to discover operations. Plaintext traits can describe how +values are represented without also owning the execution API. + +Keeping both equivalent spellings adds documentation, completion, and naming +choices without adding capability. Familiarity with an earlier API is a +migration consideration, but does not by itself justify two permanent entry +points. + +The target-directed `EncryptInto` surface can choose outputs other than +ordinary ciphertext and is retained explicitly. Its context inconsistency +remains open independently of this receiver decision. No decision about +decryption or batch method receivers is made here. + +**Sources:** [plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto`, `EncryptFrom for StackCipherText`) and +[context types](../../packages/stack-encrypt/src/target/context.rs) +(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, +`src/plaintext/mod.rs`. + +## 4. The scalar derive cannot declare a fixed literal context + +**Status:** Direction agreed; implementation pending. + +The desired combination is a derived output for a bare `i32`, a fixed +context declared on that output, and `10_i32.encrypt_into(&keyset)`. +The current macro rejects `#[stash(plaintext = i32, context = "example")]`: +the literal `context` attribute is supported only with `struct = ..`. + +A supported derive-only workaround wraps the integer in a plaintext struct: + +```rust +struct IntegerInput { + value: i32, +} + +#[derive(stack_encrypt::EncryptFrom)] +#[stash(struct = IntegerInput, context = "example")] +struct EncryptedInteger { + value: stack_encrypt::StackCipherText, +} + +// With EncryptInto imported: +let encrypted: EncryptedInteger = IntegerInput { value: 10 } + .encrypt_into(&keyset) + .await?; +``` + +This form declares the field's context as the pair `("example", "value")`, +rendered as the ZeroKMS descriptor `example/value`, and uses defaultable +`DeclaredContext` for optional caller extensions. It is not a direct +conversion from a bare integer. A manual `EncryptFrom` implementation +can supply its own context; the ordinary literal attribute cannot express +that on a scalar derive today. + +### Agreed direction + +Allow a scalar `plaintext = T` derive to declare a fixed literal context: + +```rust +#[derive(EncryptFrom)] +#[stash(plaintext = i32, context = "example/integer")] +struct EncryptedInteger { + ciphertext: StackCipherText, +} + +let encrypted: EncryptedInteger = 10_i32.encrypt_into(&keyset).await?; +``` + +With a fixed context, the generated operation supplies that context and +exposes a defaultable caller-context type, enabling `encrypt_into`. Without +a fixed context, the ordinary scalar derive continues to use `CallerContext` +and requires `encrypt_into_with_context`. Preserve explicit `context_type` +and `context_field` configurations; their interaction with a fixed context +needs validation rules rather than a silent override. + +This addresses the scalar gap at the derive layer without requiring +`CallerContext` itself to implement `Default` or changing the `EncryptFrom` +trait. A bare scalar has no input field from which the derive can extract +context; runtime context can still be supplied by the caller. + +Additional caller context is accepted. The existing mechanism to reuse is +`DeclaredContext::under(own)`, exposed on operation descriptions through +`Encryption::under(own)`. `DeclaredContext` implements `Default`; its empty +default preserves the declared context unchanged. A caller extension is +combined as `(own, caller)`, never substituted for the declared context. + +Recommended combination semantics follow this existing mechanism: + +| Caller input | Effective structured context, with static context `C` | +| --- | --- | +| No extension | `C` | +| One extension `P` | `(C, P)` | +| Successive extensions `P`, then `Q` via `DeclaredContext::with` | `((C, P), Q)` | +| One extension containing the pair `(P, Q)` | `(C, (P, Q))` | + +This is structured composition, not string concatenation. Order and grouping +matter, and decrypt must reproduce the same structure. `NonEmpty::with` +builds the corresponding pairs directly. The existing test +`under_gives_a_subtree_its_own_context_which_the_callers_extends` in +[target.rs](../../packages/stack-encrypt/tests/target.rs) verifies that an +operation's declared context plus a caller extension agrees with a directly +composed context. + +The exact representation of a static literal (notably text versus a parsed +path label) still needs to be specified when implementing the derive, so +the descriptor and AEAD context remain consistent across encryption and +decryption. This decision is separate from retaining the `EncryptInto` trait. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`a_container_context_requires_a_struct`) and +[generated implementations](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`struct_derive`, `value_derive`). + +## 5. EncryptFrom does not define encrypt_from + +**Status:** Open. + +`EncryptFrom` defines the associated type `Context`, the operation-description +constructor `encryption()`, and `indexes()`. It does not define a function +named `encrypt_from`. + +A function with that name does exist, but on the blanket extension trait +`EncryptInto`: `Target::encrypt_from(&value, &keyset, context)` delegates to +`keyset.encrypt_as`. Calling that function requires `EncryptInto` in scope +and a target implementing `EncryptFrom`. + +This differs from the standard Rust naming relationship: `From` defines +`from`, while `Into` defines `into`. The names suggest that same +relationship here, but `EncryptFrom` declares an operation rather than +executing a conversion, and the target-side execution function belongs to +the source-side extension trait. That makes method discovery and the +required trait import surprising. + +Review the trait and method names, or where the convenience function lives, +while preserving the distinction between declaring an operation and +executing it. No replacement design has been selected. + +**Source:** [target operation traits](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptFrom`, `EncryptInto`, including `EncryptInto::encrypt_from`). + +## 6. Plan builder encrypt_into reads like the execution method + +**Status:** Open; revisit during the plan API discussion. + +`Plan::value::().encrypt_into::()` is an inherent builder method that +selects an output target requiring `T: EncryptFrom`. It does not implement +`EncryptInto` or execute encryption. Sharing the name with +`value.encrypt_into(&keyset).await?` caused confusion during this review. + +Consider renaming the builder method to `target`: + +```rust +Plan::value::() + .target::() + .build() +``` + +This names the selection being made and distinguishes it from executing a +conversion. The rename is a proposal for later discussion, not a decision +or an implemented API change. `EncryptInto::encrypt_into` remains unchanged +by this proposal. + +**Source:** [value plan builder](../../packages/stack-encrypt/src/plan/value.rs) +(`ValueStart::encrypt_into`). + +## 7. Mixed encrypted and plaintext fields lack a direct output-derive spelling + +**Status:** Passthrough attribute proposal accepted in principle; implementation pending. + +For a `User` with encrypted `name` and `email` and plaintext `id` and +`created_at`, the fields-plan API directly supports `.encrypt(pick(..))` +and `.passthrough(pick(..))`. It batches the encrypted fields and returns +`FieldValues`; callers can extract those values into a typed storage struct. + +The current `EncryptFrom` derive has no passthrough/copy attribute for source +fields. `#[stash(default)]` initializes an output from a default or expression; +it does not copy a field from the input. Merely keeping the original field +type does not select passthrough either: derived fields require target traits. + +The proposed `#[stash(passthrough)]` attribute on output fields was accepted +as sensible: it copies the matching source field unchanged, alongside fields +whose types select encryption. Plaintext passthrough fields remain unencrypted and +unauthenticated; making a field part of the encryption context is a separate +choice, not implied by copying it. + +This gap is specific to Stack's output-side `EncryptFrom` derive. Vitamin C +already has plaintext-side `#[derive(Encrypt, Decrypt)]` with +`#[aead(passthrough)]` on source fields. Stack's `keyset.encrypt(&user, aad)` +can consume that derived source, yielding a generic `StackCipherText` map +with sealed and passthrough nodes. It does not generate a named +`EncryptedUser` type or expose the output as typed Rust fields. Its map-entry +AAD derivation also differs from the per-field contexts of a Stack fields +plan; the two routes should not be presented as interchangeable storage +formats. There is currently no source-side derive that generates the typed +encrypted counterpart illustrated in this review. + +**Sources:** [plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) +(`FieldsBuilder::passthrough`) and +[derive field attributes](../../packages/stack-encrypt-derive/src/attrs.rs) +(`FieldAttrs`). + +## 8. Clarify whole-value encryption, layouts, and searchable outputs + +**Status:** Open architectural discussion; recommendations below are not decisions. + +Three user workflows need clear support: + +1. Serialize a complete value into bytes and encrypt those bytes as one + opaque ciphertext. +2. Produce a chosen layout, such as `EncryptedUser`, with independently + encrypted fields and explicit plaintext fields. +3. Produce a layout with additional encrypted forms or search terms, such + as EQL targets. + +The existing Vitamin C plaintext derive produces a structural ciphertext +tree, not a single encrypted serialization. It is a generic output layout +for workflow 2, not workflow 1. The distinction affects key count, +independent field access, visible structure, and authenticated boundaries. +Encoding the completed ciphertext tree into one byte string would not +change those properties into whole-value encryption. + +### Recommended model + +Distinguish the encryption granularity (one encoded value or selected +fields) from the output representation and its requested operations. +Workflow 3 should extend workflow 2 through field targets that request +ciphertext, terms, or both, rather than introducing a separate execution +system. The generic tree can remain a layout option without being the +mandatory scalar storage representation. + +- **Whole-value path:** the follow-up review favors an explicit named + wrapper, provisionally `OpaqueBlob`, which owns serialization and + deserialization through one library-selected codec. It produces one opaque + persistable ciphertext; decrypting into `OpaqueBlob` selects the matching + decode path. Today callers can serialize into `Vec`, which Vitamin C + treats as a single byte leaf, but still encounter the node/leaf storage gap. +- **Layout path:** select a target with `EncryptFrom`/`EncryptInto`, or a + plan, with an explicit passthrough choice. A source-side derive that + generates a target would be authoring convenience over the same system, + not another set of encryption semantics. +- **Searchable layout path:** compose core encryption/index operations + through targets. EQL adapters own their storage representation; Stack + owns execution. Write and query operations must share the relevant + context and normalization rules. + +Keep source representation, requested operations, output encoding, and +execution distinct internally, while giving each workflow a short public +path. An application should not need to learn visitor machinery or plans +to encrypt and persist a scalar. Arbitrary serialization must not become an +implicit replacement for the typed normalization used by indexes. + +Before selecting more trait names, write end-to-end examples for all three +workflows: encrypt, persist, load, decrypt, plus query for searchable targets. +Use those examples to evaluate the public surface and error messages. + +### Follow-up: opinionated whole-value wrapper + +The first public example now uses +`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` rather than +manual JSON encoding. The wrapper name is provisional; the direction is to +make blob serialization an explicit library-owned behavior. The stored result +remains opaque ciphertext, and decryption returns an `OpaqueBlob` that +can be unwrapped. The codec and its authenticated versioning must be specified; +the wrapper does not silently change how unwrapped structs are encrypted. + +Constant-time serialization/deserialization is a desired property to +investigate, not a property established by choosing a binary codec or using +constant-time AEAD. The first revised example provisionally uses Serde traits, +whose arbitrary user implementations cannot carry that guarantee. Define +which lengths and shapes may be public and whether a stricter supported-type +profile is needed before promising secret-independent execution. The examples +document this unresolved boundary separately from the wrapper ergonomics. + +Investigate `serdect` as part of that work. It is already a Stack Encrypt +dependency; check its encoding/decoding guarantees and their scope, and +evaluate applicability to ciphertext serialization and the proposed blob +codec. No whole-object constant-time guarantee or codec choice is inferred +from its use. + +## 9. Ownership of the value-encryption API across Stack and Vitamin C + +**Status:** Open architectural discussion; moving the traits is proposed, +not authorized implementation work. + +The review has repeatedly crossed from Stack's target declarations into +Vitamin C's plaintext visitor traits, cipher traits, contexts, derives, and +ciphertext tree. The user proposes moving the encryption traits into Stack +Encrypt and retaining general primitives such as protected values and +randomness in Vitamin C. + +**Recommendation:** Stack should own the public value-encryption model, +including the plaintext traits/derives, target declarations, contexts, +ciphertext persistence, and execution semantics. Vitamin C can retain +reusable cryptographic and memory-protection primitives. Moving files alone +does not resolve ambiguous names (`Encrypt` currently describes a source), +duplicate entry points, or the missing whole-value/storage path. + +One Stack-owned core layer may serve Rust and language guests without +pulling in KMS transport or HTTP. Applications should still have one +coherent `stack_encrypt` entry point; an internal dependency split need not +become another public API users must understand. + +### Migration constraints + +- Today `stack-encrypt` depends on both `vitaminc-aead` and + `vitaminc-encrypt`, while `vitaminc-encrypt` also depends on + `vitaminc-aead`. Moving the latter's traits into the top-level Stack crate + and making the existing backend depend on Stack would introduce a cycle. + Move the value-level backend adapters too, or separate a lower-level + core/primitive boundary. +- Context composition, type tags, ciphertext framing, index normalization, + and binding transport frames have compatibility consequences. Preserve + their existing encodings when changing ownership; any deliberate semantic + change needs separate versioning and migration treatment. +- Retain one canonical trait/type identity. Compatibility re-exports can + preserve an old path where dependencies permit; copied independent trait + definitions would require duplicate implementations and adapters. +- Account for guest/FFI consumers and EQL target implementations, not just + Rust application imports. The no-HTTP/WASM path must remain usable. + +**Sources:** [Stack dependencies](../../packages/stack-encrypt/Cargo.toml), +[guest ABI dependencies](../../packages/stack-guest-abi/Cargo.toml), and +[EQL encryption adapters](../../packages/eql/crates/eql-bindings/src/encryption.rs). +The installed `vitaminc-encrypt` 0.8.0 manifest confirms its dependency on +`vitaminc-aead`. + +## 10. Public encrypt_as duplicates encrypt_into + +**Status:** Decision agreed; implementation pending. + +`KeysetCipher::encrypt_as` is currently an inherent method requiring the +output to implement `EncryptFrom`. It executes that target's operation +declaration. `EncryptInto::encrypt_into` and +`EncryptInto::encrypt_into_with_context` delegate to it, supplying either +the default context or the caller's context. + +**Decision:** remove `encrypt_as` from the public API. Keep `encrypt_into` +and `encrypt_into_with_context` as the public target-directed entry points: + +```rust +let encrypted: EncryptedInteger = value.encrypt_into(&keyset).await?; +let encrypted: EncryptedInteger = value + .encrypt_into_with_context(&keyset, context) + .await?; +``` + +The operation-execution machinery may remain internal; removing the public +method does not remove target declarations, batching, or execution. Public +examples should use the retained entry points when illustrating the proposed +API. Earlier descriptions of `encrypt_as` in this log describe current code. + +This is distinct from issue 3's decision about ordinary encryption: +`keyset.encrypt(value)` remains the sole ordinary `encrypt` spelling, while +`value.encrypt_into(&keyset)` selects a particular output type. The +`encrypt_from` naming/location question in issue 5 remains open. + +**Source:** [target operations](../../packages/stack-encrypt/src/target/operations.rs) +(`KeysetCipher::encrypt_as`, `EncryptInto`). + +## 11. Context-field storage does not yet extract context from an input struct + +**Status:** Open. + +`#[stash(context_field)]` exists on derived output types. On encryption it +stores the supplied context in that output field; on decryption it uses the +stored context, optionally checking an independently supplied expected value. +It does not automatically select a field from a plaintext input struct. + +The macro currently treats this field as the complete context and rejects +it alongside a literal `context`. Since `struct = User` requires a literal +context, that form cannot currently mark its `id` with `context_field`. +The public examples originally described passing the id explicitly without +explaining this distinction. + +### Clarified model: context source and context storage are independent + +Two similar scenarios must be distinguished: + +1. The caller supplies context as a function argument; the encrypted output + stores it for later use. This is the existing scalar-output + `context_field` behavior. +2. The encryption declaration obtains context from one or more fields of + the plaintext. Whether those values are also stored is a separate choice. + +The second scenario can include storage too: read `User.id`, use it to bind +the encrypted fields, and carry it into `EncryptedUser.id`. The caller need +not pass the same value twice. Several fields, such as a tenant id and user +id, can supply a structured composite context. + +| Context source | Stored in the encrypted output? | Recovery requirement | +| --- | --- | --- | +| Caller argument | Yes | Read the stored context; optionally check an independently supplied expected value | +| Caller argument | No | Supply the same context again | +| Plaintext field(s) | Yes | Reconstruct context from the stored field(s); optionally check expected values | +| Plaintext field(s) | No | Supply equivalent context independently before decrypting | + +Store a source-derived context independently of the ciphertext that needs +it, so decrypt does not depend on first opening that ciphertext to discover +its own context. Binding ciphertext to a stored context authenticates that +relationship; it does not by itself establish that the entire record is at +the destination the application expected. + +The derive should express source selection and output storage as separate +concepts, even if a shorthand covers the common read-and-store case. Avoid +assuming that one `context_field` attribute should silently perform both. +Syntax remains open. Define field ordering/grouping for composite context, +literal prefixes, optional caller extensions, `from` mappings, and expected +context checks. These capabilities are proposed, not implemented. + +The recommended common-case shorthand is `context(field = "id")`, expanding +exactly to `encrypt(field = "id"), decrypt(field = "id")`. It selects the +same Rust field name on the two different inputs; it does not implicitly +copy or store that field. Keep the directional form for different names or +sources. Initially reject mixing shorthand and directional selectors to +avoid ambiguous precedence. The context examples now use this shorthand +for the plaintext-id-to-stored-id case. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`Record::parse`, `context_type`) and +[context-field output assembly](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`assignments`, `value_derive`). + +## 12. Default is not the same as optional caller context + +**Status:** Open; exposed by the directional-context examples. + +`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` make the +no-context form available whenever the relevant associated `Context` type +implements `Default`. That works for deliberately defaultable types such as +`DeclaredContext` and `ExpectedContext`, and currently excludes +`CallerContext` and `AeadContext`. + +However, a required caller value can have a Rust default: if a declaration +exposes `Context = u64`, the no-context form supplies zero. The directional +proposal `encrypt(caller = u64)` must not accidentally mean "optional, with +zero as the default". + +Separate the application's context value type from the trait's caller-input +type or capability. Options include a non-defaultable `RequiredContext` +wrapper that accepts ergonomic conversions from `T`, or a dedicated marker +controlling context-free entry points. No mechanism is decided yet. + +This matters equally to required context on decrypt. Source-derived context +can require no encrypt argument, while decrypt may require an external +value or accept an optional expectation of stored context. Do not assume the +two associated input types must match, or that either always equals the +complete logical context used by encryption. + +See the [context scenario examples](2026-10-24-stack-encrypt-context-examples.md) +for the proposed selectors and input requirements. + +**Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md new file mode 100644 index 000000000..c927b7d67 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -0,0 +1,723 @@ +# Context sources and storage: directional API examples + +Date: 2026-10-24. Design exploration accompanying the +[API review log](2026-10-24-stack-encrypt-api-review.md) and +[complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md). + +**Everything introduced here is proposed syntax, not an implemented API.** +The purpose is to compare context behavior across the meaningful source and +storage combinations before selecting attributes or changing traits. The +`Ciphertext` and `passthrough` proposals are inherited from the workflow draft. + +## Recommendation: directional sources, separate storage + +Borrow Serde's nested directional attribute style: it supports separate +serialization and deserialization names within one `rename` attribute. +See [Serde's field attributes](https://serde.rs/field-attrs.html). + +For encryption, use direction names to say **where to obtain context**: + +```rust +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(field = "id") +))] +``` + +The two `field` selectors refer to different objects: the plaintext input +when encrypting, and the encrypted input when decrypting. They must yield the +same typed context value. Unlike a Serde rename, these are not independent +cryptographic names: changing the context value or its shape between the two +directions prevents decryption. + +For the common case where both directions use the same Rust field name, +the recommended shorthand is: + +```rust +#[stash(context(prefix = "users", field = "id"))] +``` + +This expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. +It still reads two different objects: the plaintext input on encrypt and the +encrypted input on decrypt. It does not copy or store the field implicitly; +the output layout must make that value available, normally with passthrough. +Use the explicit directional form when the names or sources differ. + +Use a separate field annotation to **store the computed context** when it is +not already carried by ordinary passthrough fields: + +```rust +#[stash(from_context)] +scope: u64, +``` + +`from_context` fills an output metadata field from the computed logical +context. It does not choose the context source. A passthrough source field +can already carry the context value, in which case no `from_context` field +is needed. No new source-side encryption methods or public `encrypt_as` +entry point are introduced. + +This is preferable to bare `context(in)` / `context(out)`: either term could +mean an operation direction, an input/output object, or storing a value. +`encrypt`, `decrypt`, and `from_context` give those roles distinct names. + +## Proposed vocabulary and semantics + +| Syntax inside `context(...)` | Meaning | +| --- | --- | +| `prefix = "users"` | Static base for this layout's field contexts | +| `field = "id"` | Shorthand for both `encrypt(field = "id")` and `decrypt(field = "id")` | +| `encrypt(caller = u64)` | Obtain the logical context from a required `u64` argument | +| `encrypt(field = "id")` | Obtain it from the plaintext input's `id` | +| `decrypt(caller = u64)` | Obtain it from a required `u64` argument on decrypt | +| `decrypt(field = "scope")` | Obtain it from the encrypted input's cleartext `scope` | +| `encrypt(tuple(field = "tenant_id", field = "id"))` | Build one ordered, typed pair from plaintext fields | +| `decrypt(tuple(field = "id", caller = u64))` | Build one pair from a stored field and the decrypt argument | + +`tuple(...)` is a declarative selector, not a Rust expression or a function +that gets access to keys. It can nest. `field` names resolve at compile time; +the first design supports direct fields, not arbitrary paths or callbacks. +There is at most one caller argument selector per direction; use a tuple or +typed context object as that argument when it carries several values. + +Use either the shared `field` shorthand or both explicit directional +selectors. For the initial API, reject mixing the shorthand with directional +selectors instead of inventing precedence rules. That keeps the expansion +predictable; a record with asymmetric names uses the explicit form. +`context = "users"` +remains the existing simple spelling for a static prefix with optional caller +extensions; it is used in scenario 0. + +### Composition rule + +Call the logical row context `R`. For a field named `name`, its static base is +`B = ("users", "name")` and its effective context is `(B, R)`. With only a +static prefix and no extension it is just `B`, not `(B, ())`. + +The selector's result is **one structured value**. A pair `(tenant_id, id)` +produces `(("users", "name"), (tenant_id, id))`, not +`((("users", "name"), tenant_id), id)`. `from_context` stores `R`, not the +field-dependent effective context. Static prefixes and field identities are +reconstructed from the declaration. + +Names in this proposal do not change existing byte encodings. Mapping new +selectors onto the existing context machinery must preserve types, order, +grouping, and descriptor rendering. A new explicit selector is a new +declaration; it must not silently reinterpret existing stored data. + +### Storage and expected context + +When all of `R` can be reconstructed from stored fields, the default decrypt +call uses those fields. The caller may additionally supply an expected `R` +through the existing `ExpectedContext` pattern: check equality first, then +use the stored value. That expected value is **not another extension**. + +When a decrypt selector needs a caller argument, that argument is required +to construct `R`; no no-context convenience is available. It is not an +expected-value check on context already fully available from the record. + +If a future caller needs both a required input and a separate expectation +on the assembled `R`, use a distinct checked-context input or a named API. +Do not overload one unlabelled argument to mean both. That combined check +API remains outside this proposal; scenario 7 makes the need visible. + +## Scenario matrix + +| Scenario | Encrypt obtains context from | Context available in stored output | Decrypt obtains context from | +| --- | --- | --- | --- | +| 0 | Static prefix; no row context | Declaration | Declaration | +| 1 | Caller argument | Not stored | Caller argument | +| 2 | Caller argument | Dedicated metadata field | Metadata field | +| 3 | Plaintext `id` | No cleartext copy; `id` itself is encrypted | Caller argument | +| 4 | Plaintext `id` | Passthrough `id` | Stored `id` | +| 5 | Plaintext `(tenant_id, id)` | Two passthrough fields | Same pair of stored fields | +| 6 | Plaintext `id` plus caller scope | Dedicated pair-valued metadata field | Stored pair | +| 7 | Plaintext `id` plus caller scope | Only `id` is stored in cleartext | Stored `id` plus caller scope | +| 8 | Plaintext `(tenant_id, id)` | Neither component is stored in cleartext | Caller supplies the pair | + +An encrypted copy of a context field does not make that context available +before decrypt. Scenarios 3 and 8 retain encrypted copies to recover the +complete plaintext object; callers still need the context independently. + +## Shared code for the examples + +Each scenario defines an output type and an async function. Combine this +prelude, the scenario blocks, and the final `main` into one candidate program. +They use real file serialization through the same helper, so no example +relies on a context value being secretly carried inside an in-memory handle. +All use proposed APIs and therefore cannot yet be typechecked against Stack. + +Dependencies match the workflow draft: Stack Encrypt, Serde with derives, +serde_json, tempfile, and Tokio. ZeroKMS credentials are required once these +APIs exist. `scope = 7` in caller examples is external application context, +not a keyset selector or an authorization credential. + +```rust +use serde::{de::DeserializeOwned, Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, NonEmpty, + StackCipher, +}; +use std::error::Error; + +type Result = std::result::Result>; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_user() -> User { + User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +// The candidate Ciphertext Serde contract encodes its canonical envelope. +fn store_and_reload(value: &T) -> Result { + let directory = tempfile::tempdir()?; + let path = directory.path().join("record.json"); + std::fs::write(&path, serde_json::to_vec(value)?)?; + Ok(serde_json::from_slice(&std::fs::read(path)?)?) +} +``` + +## 0. Static prefix only + +This is the baseline. There is no dynamic row context to store or supply. +It retains the existing static-context form and proposed passthrough support. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct StaticUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_0() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StaticUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +For this shorthand, caller extensions remain available as previously agreed: +encrypt and decrypt must receive the same extension. The remaining examples +use explicit directional selectors, so their argument requirements follow +those selectors; there is no additional implicit extension layered on top. + +## 1. Caller context, not stored + +`scope` is absent from the output. The caller must provide it again. A +plaintext `id` in this layout is unrelated to the chosen context source. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(caller = u64) +))] +struct ExternalScopeUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_1() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: ExternalScopeUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context: `7_u64`. Effective name context: `(("users", "name"), 7_u64)`. + +## 2. Caller context, stored in an output metadata field + +The source is still the function argument. `from_context` stores its value; +the decrypt selector explicitly chooses that stored field. `scope` is output +metadata, not a field to add to the recovered `User`. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(field = "scope") +))] +struct StoredScopeUser { + #[stash(from_context)] + scope: u64, + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_2() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredScopeUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.scope, 7); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context remains `7_u64`. It is stored once, not appended a second time +on decrypt. This generalizes the existing caller-supplied-and-stored case to +the field-by-field layout. + +## 3. Plaintext field as context, no cleartext context stored + +The encrypt selector reads `User.id`; no argument is needed. The output's +`id` is encrypted, so it cannot supply the context required to open itself. +The reader obtains the id from its external record lookup. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(caller = u64) +))] +struct HiddenIdUser { + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_3() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let lookup_id = user.id; + + let stored: HiddenIdUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, lookup_id).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `R = 42_u64`. Supplying a different id fails to +open the encrypted fields. The id is not dropped or defaulted when rebuilding +`User`; its encrypted copy is opened using the externally supplied context. + +## 4. Plaintext field as context, stored as the corresponding output field + +This is the natural ID-bound form of workflow example 2. The context selector +and the passthrough annotation have separate jobs: select `id`, and copy it. +It uses the common-field shorthand, equivalent to the explicit pair of +directional selectors introduced above. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context(prefix = "users", field = "id"))] +struct IdBoundUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_4() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: IdBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + + // Alternatively, require the stored context to match the lookup's id. + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context( + loaded, + &cipher, + NonEmpty::from(user.id), + ) + .await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The second decrypt call uses the `ExpectedContext` conversion. It checks +the stored id instead of constructing `(stored_id, expected_id)`. Changing +`id` alone breaks decryption; moving an entire consistent record is detected +only when the reader supplies the expected id (or another placement check). +`created_at` remains ordinary unauthenticated passthrough metadata. + +## 5. Several plaintext fields as context, stored in separate fields + +Add a tenant id to the input. The context is the ordered pair `(tenant_id, id)`. +The output renames `id` to `user_id` to demonstrate why direction-specific +field selectors are useful. No duplicate context metadata field is needed. + +```rust +#[derive(Debug, PartialEq)] +struct TenantUser { + tenant_id: u64, + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_tenant_user() -> TenantUser { + TenantUser { + tenant_id: 7, + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(tuple(field = "tenant_id", field = "user_id")) +))] +struct TenantBoundUser { + #[stash(passthrough)] + tenant_id: u64, + #[stash(passthrough, from = "id")] + user_id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_5() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + + let stored: TenantBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = TenantUser::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `(7_u64, 42_u64)`. Rust field order does not choose +the context order: the selector does. A Serde wire rename also does not change +the selector; it names Rust fields, not serialized property names. + +## 6. Plaintext field plus caller argument, store the combined context + +Here `R = (User.id, caller_scope)`. `binding` stores that complete pair. The +original id is also retained as ciphertext so `User` can be reconstructed +without adding a new rule that maps pieces of metadata into plaintext fields. +That deliberately redundant representation keeps this example focused on +context composition rather than projection conveniences. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(field = "binding") +))] +struct StoredCombinedUser { + #[stash(from_context)] + binding: (u64, u64), + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_6() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredCombinedUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.binding, (42, 7)); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The encryption function accepts only the caller-supplied component (`7`), not +the pair the derive can construct itself. A checked decrypt would expect the +whole reconstructed pair `(42, 7)`, not just that caller component. + +## 7. Plaintext field plus caller argument, store only the field component + +Store `id` as a normal field, but keep the external scope out of the output. +Decryption combines the stored id with the scope supplied by the reader. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(tuple(field = "id", caller = u64)) +))] +struct PartiallyStoredUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_7() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: PartiallyStoredUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This also constructs `(42_u64, 7_u64)` in both directions. It shows why +"stored context" cannot always be represented as a single yes/no flag: +some components may be stored and others supplied externally. A separate +expected-id check can be performed against `loaded.id` before opening when +the application also needs to assert the record's destination. + +## 8. Several plaintext fields as context, keep neither component in cleartext + +The encrypt side obtains `(tenant_id, id)` from the source. The reader supplies +the complete pair from outside the encrypted record; both source fields are +recovered from ciphertext after that context has been established. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(caller = (u64, u64)) +))] +struct HiddenTenantUser { + tenant_id: Ciphertext, + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_8() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + let lookup_context = (user.tenant_id, user.id); + + let stored: HiddenTenantUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = + TenantUser::decrypt_from_with_context(loaded, &cipher, lookup_context).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This is a new typed selector contract. It does not mean flattening the pair +into text, or treating separately chained extensions as interchangeable with +one tuple-valued argument. + +## Candidate program entry point + +```rust +#[tokio::main] +async fn main() -> Result<()> { + example_0().await?; + example_1().await?; + example_2().await?; + example_3().await?; + example_4().await?; + example_5().await?; + example_6().await?; + example_7().await?; + example_8().await?; + Ok(()) +} +``` + +## Consequences for the traits and derive + +The logical context `R` is not always the value a caller passes. For example: + +| Scenario | Logical `R` | Caller input on encrypt | Caller input on decrypt | +| --- | --- | --- | --- | +| 1 | `u64` | Required `u64` | Required `u64` | +| 2 | `u64` | Required `u64` | Optional expectation of `u64` | +| 3 | `u64` | None | Required `u64` | +| 4 | `u64` | None | Optional expectation of `u64` | +| 6 | `(u64, u64)` | Required `u64` component | Optional expectation of the whole pair | +| 7 | `(u64, u64)` | Required `u64` component | Required `u64` component | +| 8 | `(u64, u64)` | None | Required pair | + +Today `EncryptFrom::Context` and `DecryptInto

::Context` already belong +to separate traits and can differ. They describe the input the caller needs +to supply, not necessarily the final context used by every operation. That +distinction should be explicit in documentation; `ContextInput` might be a +clearer name, but a rename is not decided here. + +No-caller-input selectors can expose a defaultable input, such as `()`, to +support `encrypt_into`. Stored-context decrypt uses the existing +`ExpectedContext` idea. Required caller selectors must remain required +even if their Rust type implements `Default`: a required `u64` must not +silently become zero merely because `u64: Default`. + +That exposes a real constraint in today's convenience methods, whose only +test is `Context: Default`. An implementation of this proposal would need +a non-defaultable `RequiredContext` input (with an ergonomic conversion +from `T`), or a separate trait capability controlling context-free calls. +The examples show bare caller values at call sites, but do not assume that +the associated input type can be bare `u64`. Choose that mechanism before +claiming the directional API enforces required inputs. + +The current public `Encryption` constructors must also gain or expose the +core-owned source-selection machinery needed by the derive. Existing plan +context fields demonstrate source-dependent context, but are label-based, +carry the field, and use different composition rules. They are not already +an implementation of this proposal's general typed, optionally stored `R`. + +## Rules the implementation should enforce + +- Selectors for both directions must produce the same context type and + ordered structure. Different Rust field names are fine; different values + or grouping are not a supported way to encrypt and decrypt one record. +- Resolve field names and types at compile time. A decrypt selector cannot + read an encrypted field to obtain the context needed to decrypt it. +- A `from_context` field is cleartext output metadata with the logical + context's type. It cannot also be encrypted, defaulted, or independently + copied from an unrelated source. Ordinary passthrough fields do not + become context merely because their names are `id` or `tenant_id`. +- Extra metadata is not invented as a plaintext field during recovery. + Every original plaintext field must have a defined recovery path; no + silent omission or defaulting to compensate for a context-only source. +- Check supplied expectations before key retrieval where possible. A + context value read from storage remains untrusted until encrypted fields + authenticate under it. An expectation check alone is not authentication. +- If a selected source value is duplicated in stored context metadata and + in a separately recovered plaintext field, validate their agreement + before returning the reconstructed object. Do not return a different id + from the one the encrypted fields were bound to. +- Stored context should be required in the serialized schema; a missing + field must not silently fall back to a default context. Empty-context + policy must be explicit and retain the current nonempty target rule. +- For indexed targets, the selected context also scopes their terms unless + a future API explicitly separates index context from ciphertext context. + A row-specific id therefore changes search semantics; these examples + concern ciphertext layouts and do not silently promise cross-row queries. + +Required tests include storage round trips for every scenario, missing +required arguments, wrong expected values, altered stored context, tuple +reordering/regrouping, renamed source/output fields, and attempted circular +decrypt dependencies. Same-typed field swaps may evade a type check, so +round-trip and mismatch tests remain necessary even with typed selectors. + +## Alternatives worth comparing + +1. **Field-level `context(encrypt, decrypt)` flags.** Concise for one id, + but references the output declaration while the encrypt source lives + elsewhere, leaves multi-field order implicit, and cannot readily describe + caller-plus-field composition. Prefer explicit container selectors first. +2. **One `context = expression` callback.** Flexible, but hides dependencies, + makes reverse reconstruction and cycle checks harder, and encourages + arbitrary application code in the declaration. Keep simple declarative + selectors as the initial surface. +3. **A typed context object with a dedicated derive.** Could reduce repetition + for large composite contexts and carry a stronger encoding contract. + Consider it after the scalar/tuple cases establish the desired semantics. + +The recommendation is the directional selector plus independent storage +model, not a commitment to every spelling shown here. In particular, +`from_context`, `prefix`, the tuple notation, and how required inputs are +represented remain review choices. + +## Evidence and validation + +Current behavior was checked in +[context types](../../packages/stack-encrypt/src/target/context.rs), +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[derive validation](../../packages/stack-encrypt-derive/src/shape.rs), and +[plan context fields](../../packages/stack-encrypt/src/plan/build.rs). +The directional nesting is inspired by Serde, not provided by Serde. + +These are candidate API examples. Syntax/link checks do not establish macro +support, type correctness, or encryption behavior. No live encryption or +database tests are claimed, and no library implementation is changed. + +Validation completed: `rustfmt` parsed the combined nine-scenario candidate +program and the introductory fragments (wrapped as Rust items), using edition +2024. Local links, Markdown fences, and trailing whitespace were checked in +this document and both companion documents. No proposed macro was expanded +or typechecked. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md new file mode 100644 index 000000000..da361fc04 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -0,0 +1,572 @@ +# Stack Encrypt: complete public API examples for review + +Date: 2026-10-24. Companion to the +[API review log](2026-10-24-stack-encrypt-api-review.md). +The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) +explore directional source selection, storing context, and decrypt inputs +across nine scenarios. Their attributes are proposals, not implemented APIs. + +This document tests the proposed public API against three complete workflows: +encrypt, persist, load, and decrypt a user; the searchable workflow also +constructs and applies a query. It is an API design draft, not documentation +for a released version. The examples deliberately leave plans and operation +descriptions out of the application code. + +Each main example is a separate program, with all imports and types shown. +They use the same user fields and synthetic data. `created_at` is an `i64` +Unix timestamp in seconds so that date-library configuration does not distract +from the encryption API. The storage files live in a temporary directory and +are removed when the program exits. Files stand in for a database boundary; +the examples really encode, write, read, and decode their stored forms. + +## What is agreed, and what these examples propose + +| Surface | Status in this review | +| --- | --- | +| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | +| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | +| No public `encrypt_as` in these examples | Agreed direction; current implementation still exposes it | +| `#[stash(passthrough)]` on output fields | Accepted in principle; needs implementation on both encrypt and decrypt derives | +| A fixed context on a scalar output derive | Agreed direction; not required by these particular examples | +| `OpaqueBlob` | Agreed wrapper direction; provisional name. Requests whole-value serialization and one encrypted leaf, using a library-owned codec | +| `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | +| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | +| `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | +| `Ciphertext` as an encryption/decryption target | Proposed scalar target, including support for `String` and `Vec`; does not silently flatten a struct into a blob | +| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | +| `cipher.decrypt_with_context` | Proposed companion to `encrypt_with_context`; ordinary decrypt naming remains a review decision | +| `User::decrypt_from` / `decrypt_from_with_context` | Existing `DecryptFrom` extension conveniences, retained here pending the decrypt naming discussion | +| `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | + +The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or +passthrough APIs. They are complete candidate programs, not programs claimed +to compile against today's crate. The names and contracts above remain +subject to review where marked proposed or provisional. + +The proposed opaque ciphertext is intentionally untyped: the decrypt call's +expected type selects decoding. A typed wrapper remains an open alternative. +Its byte decoder parses a stored envelope; authentication happens on decrypt. +For a derived field it must preserve the existing context and leaf encoding, +not introduce a new cryptographic format just to improve ergonomics. + +The trait name `DecryptFrom` is somewhat surprising: it supplies the +convenience functions, while `#[derive(DecryptInto)]` describes how an +encrypted output recovers plaintext. These examples keep that existing +distinction visible rather than silently redesigning it. + +## Setup shared by the programs + +The proposed programs use `stack-encrypt` with its default `http` feature, +`serde` with `derive`, `serde_json`, `tempfile`, and `tokio` with `macros` and +`rt-multi-thread`. No published version currently supplies all proposed APIs, +so this draft does not give a misleading installable Cargo manifest. + +`StackCipher::new()` uses the configured ZeroKMS credentials: a CLI login or +`CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. +The second client in each program represents a later reader using the same +authorized configuration. The default keyset is selected for brevity; an +already-selected tenant keyset can be used in its place. + +Decryption uses the client because the sealed value carries its keyset id. +It still requires access to the corresponding keys and the correct context. +The examples do not depend on in-memory fake keys surviving a reload. + +## 1. Encrypt the whole user through an explicit blob wrapper + +**Intent:** store one encrypted document. All four fields, including the ID +and creation time, are inside one encrypted serialization. There are no +independently readable fields or search terms in the stored ciphertext. + +`OpaqueBlob::new(user)` explicitly requests that treatment. Stack owns the +serialization and matching deserialization, rather than requiring every +application to select and invoke a codec. The proposed wrapper owns the input; +`new` only wraps it, and serialization errors surface from the encryption +operation. The context identifies the application's document schema and binds +this document to an externally known record id. + +For this draft the wrapper accepts Serde `Serialize` on write and +`DeserializeOwned` on read. That input-trait choice is provisional: it does +not claim that arbitrary Serde implementations can satisfy a constant-time +contract. The initial wrapper has one library-selected, versioned binary +codec, not a caller-selected codec parameter. Selecting and specifying that +codec remains implementation/design work; no particular binary format is +claimed to be implemented or constant-time here. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{nonempty, Ciphertext, OpaqueBlob, StackCipher}; +use std::error::Error; + +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let expected_id = 42_u64; // Supplied by the application's record lookup. + let user = User { + id: expected_id, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + let expected = user.clone(); // Retained only for the round-trip assertion. + + // The wrapper requests one blob and lets Stack handle serialization. + let sealed: Ciphertext = keyset + .encrypt_with_context( + OpaqueBlob::new(user), + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + + // Persist the complete ciphertext envelope as binary bytes. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.ciphertext"); + std::fs::write(&path, sealed.to_bytes())?; + + // Later: reconstruct the ciphertext from storage using a new client. + let reader = StackCipher::new().await?; + let stored_bytes = std::fs::read(&path)?; + let loaded = Ciphertext::from_bytes(&stored_bytes)?; + + // Stack authenticates/decrypts first, then decodes the blob into User. + let opened: OpaqueBlob = reader + .decrypt_with_context( + loaded, + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + let restored = opened.into_inner(); + assert_eq!(restored, expected); + Ok(()) +} +``` + +**What the API must guarantee:** the serialized bytes are one encrypted leaf +under one data key. The envelope contains the retrieval metadata, so the +caller never needs to save an IV or key tag separately. A different context +must fail to open it. Stack owns the matching encode/decode pair and its +compatibility contract. A blob-format discriminator and version must be +authenticated, for example inside the encrypted payload, so decoding does +not rely on guessing a codec or on unauthenticated format metadata. This +can reuse the canonical outer ciphertext envelope; the blob payload is a +new, explicit format rather than silently reinterpreting previously encrypted +raw bytes. Application schema evolution remains explicit: a codec version +does not automatically migrate changes to `User`. + +**Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` +form. A matching no-context `reader.decrypt(loaded)` is proposed for symmetry; +it is not the signature of today's two-argument decrypt method. This document +uses explicit context in the full example so the saved format is unambiguous. + +**Meaning of the wrapper:** `OpaqueBlob` is the plaintext-side request +and the decoded value, not the stored ciphertext type. `Ciphertext` remains +the persisted envelope. `into_inner()` returns an ordinary `User`. The +wrapper does not by itself promise to zeroize arbitrary input/returned types; +the codec should keep temporary plaintext buffers in wipeable storage. +`#[derive(Encrypt)]` on `User` currently means a structural ciphertext tree +and is not a substitute for wrapping the value as one blob. + +### Timing contract to investigate + +The desired property is serialization/deserialization whose timing does not +depend on secret contents, under an explicit policy for lengths and shape. +Constant time does not mean identical elapsed time for differently sized +messages; what may vary must be declared public. The general requirement is +secret-independent control flow and memory access, including outside the +cryptographic primitive. See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). + +A library-owned binary codec gives us control over that implementation, but +does not establish the property by itself. Strings, collection lengths, +enum variants, validation errors, allocations, and user-defined Serde code +can all introduce value-dependent work. Supporting arbitrary `Serialize` / +`Deserialize` implementations cannot guarantee constant-time behavior for +the whole operation. + +Before claiming such a guarantee, define supported types, permitted public +shape/length information, and restrictions on custom implementations. A +stricter profile may need bounded shapes, fixed-width encodings, padding, +and controlled encoders/decoders, with implementation-level verification. +Padding ciphertext to a fixed size alone does not make the preceding +serialization constant-time. Authentication must complete before decoding +the plaintext blob. This is a design requirement to investigate, not a +guarantee made by these example programs or a constant-time audit result. + +**Investigation candidate: `serdect`.** It is already a dependency of +`stack-encrypt`. Review its supported encodings and precise constant-time +guarantees, including decoding/error behavior and length assumptions. Assess +where it can help with ciphertext/byte serialization and whether any of its +techniques apply to the `OpaqueBlob` codec. Do not assume guarantees for byte +encoding extend to serializing arbitrary objects through Serde. This is a +follow-up investigation, not a completed assessment or codec selection. + +## 2. Encrypt selected fields into a chosen output layout + +**Intent:** store an ordinary record with plaintext `id` and `created_at`, +and independently encrypted `name` and `email`. The output type states the +layout. The two encrypted fields need no search indexes. + +The proposed ciphertext Serde implementation makes persistence compositional: +ordinary Serde derives on the output encode plaintext fields normally and +ciphertext fields as Base64 envelopes. There is no manual `Node` traversal. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let user = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + // The output type selects the conversion and declares its own contexts. + let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + assert_eq!(encrypted.id, user.id); + assert_eq!(encrypted.created_at, user.created_at); + + // Persist the typed output, including both plaintext and ciphertext fields. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: deserialize the same output layout and recover the plaintext type. + let reader = StackCipher::new().await?; + let loaded: EncryptedUser = serde_json::from_slice(&std::fs::read(&path)?)?; + let restored: User = User::decrypt_from(loaded, &reader).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +**Context and execution:** `name` uses the structured context +`("users", "name")`, and `email` uses `("users", "email")`. The keyset is +shared, but each encrypted field receives its own data key. One conversion +batches those key requests. The decrypt derive reconstructs `User`, opening +the ciphertext fields and copying the passthrough fields back. + +**Plaintext semantics:** copying `id` does not automatically bind the sealed +fields to it. Both plaintext fields remain unencrypted and unauthenticated. +That is the requested passthrough behavior, not a promise of whole-record +integrity. The example's name/email contexts distinguish columns, not rows. + +If the application wants row binding, it can use the existing +`user.encrypt_into_with_context(&keyset, expected_id)` entry point and the +matching `User::decrypt_from_with_context(loaded, &reader, expected_id)`. +The effective contexts then become `(("users", "name"), expected_id)` and +`(("users", "email"), expected_id)`. The reader supplies the expected id +from its record lookup. That is an explicit variation, not an automatic +effect of the passthrough attribute. + +**Existing `context_field` support:** the derive does have +`#[stash(context_field)]`, but its current behavior is to store the +caller-supplied context in an output field and use it again when decrypting. +It does not read an id from the plaintext `User`. It also supplies the complete +context: the macro rejects combining it with a literal `context = "users"`. +Because the current `struct = User` form requires that literal, the attribute +cannot currently be used on example 2's field-by-field declaration. + +**Separate source from storage:** obtaining context from plaintext fields +and carrying context into the encrypted output are independent choices. +An extended derive could read `User.id`, combine it with the declared field +contexts, and also copy it into `EncryptedUser.id`. That would enable row +binding without passing the id twice. The same model should accommodate +several source fields, such as `(tenant_id, id)`, with explicit ordering and +grouping. + +Alternatively, context can be passed as an argument and stored, passed and +not stored, or derived from plaintext fields without being stored. In either +non-stored case, the reader needs equivalent context from elsewhere before +decrypting. Source selection and storage should be separately expressible; +whether one annotation can conveniently request both remains a syntax choice. +Today's `context_field` only covers the caller-supplied-and-stored case. + +For the source-derived-and-stored case, a reader could use the stored id, +with an optional independently supplied expected id to reject a whole record +moved to another destination. Changing the stored id alone must fail +decryption. The values needed to reconstruct context must be available +without first opening the ciphertext that requires them. + +**Review point:** the declaration is concise, but requires an explicit +`EncryptedUser` type. A future source-side derive could generate that type; +it should generate this same layout and context behavior. Also decide whether +`User::decrypt_from(...)` is the desired public decrypt spelling before +adding a competing no-context `loaded.decrypt_into(...)` convenience. + +## 3. Use the same layout model with a searchable email target + +**Intent:** keep the same plaintext fields, encrypt the name normally, and +store the email's ciphertext plus an equality term. Search terms come from +the original typed plaintext through their declared operation, not from the +JSON encoding used to persist the output. + +`SearchableEmail` is an application-defined scalar target. It uses the caller +context supplied by the outer record: `("users", "email")`. It does not +declare another fixed context or append another `email` component. The +`ciphertext` and `equality` member names are output layout, not additional +cryptographic context segments. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::{ + nonempty, Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, + StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct SearchableUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let alice = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + let bob = User { + id: 43, + name: "Bob".to_owned(), + email: "bob@example.com".to_owned(), + created_at: 1_700_000_100, + }; + + // One collection conversion batches key generation across both users. + let users = vec![alice, bob]; + let encrypted: Vec = users.encrypt_into(&keyset).await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("users.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: a query uses the same keyset, field context, and equality operation. + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let wanted_email = "alice@example.com".to_owned(); + let query: EqualityTerm = wanted_email + .encrypt_into_with_context( + &query_keyset, + nonempty!("users").with("email"), + ) + .await?; + + // A file-backed comparison demonstrates matching without a SQL adapter. + let loaded: Vec = + serde_json::from_slice(&std::fs::read(&path)?)?; + let matches: Vec = loaded + .into_iter() + .filter(|row| row.email.equality == query) + .collect(); + assert_eq!(matches.len(), 1); + + // The query term cannot decrypt. Open the matched ciphertext rows in a batch. + let restored: Vec = Vec::::decrypt_from(matches, &reader).await?; + assert_eq!(restored.as_slice(), &users[..1]); + Ok(()) +} +``` + +The filter is an in-memory scan over persisted encrypted records, not a +claim of database index use. It demonstrates the entire query contract: +derive the term, compare it with the stored term, and open the matched +ciphertext. An equality match is not authentication of the stored record; +the decrypt operation authenticates its encrypted fields. Passthrough +semantics are the same as in example 2. + +Collection conversion is already supported by Stack's blanket +`EncryptFrom> for Vec` implementation when `T: EncryptFrom` +and its context supports cloning. It collects every row's pending operations +before dispatch: these two users require four data keys (name and email for +each), issued through one batched `generate_keys` call. Equality terms use +the current local index-key backend. This count concerns encryption work, +not client initialization or authentication. + +One logical `generate_keys` call is not an unlimited single HTTP request: +the KMS client chunks batches according to its configured key-count limit. +This four-key example fits in one normal request; larger collections may +span several requests without changing the public call. + +The matching collection decrypt implementation batches retrieval across +rows, with one `retrieve_keys` call per keyset represented in the ciphertext. +An empty collection requires no data-key call. Collection order is preserved +by the conversion; no new whole-collection authentication is implied. + +This target-directed vector is a vector of independent output records. It +is distinct from encrypting a plaintext vector into a single +`StackCipherText` sequence tree, and from wrapping the vector in `OpaqueBlob` +to encode/encrypt the whole collection as one value. Batching is a property +of execution, not a change to the chosen layout. + +This example uses the common field context for all rows so one query term +can search them. Adding a row id to that context would also scope the term +to that row. Separating ciphertext row binding from column-wide search +context is a further design question, not something these examples silently +promise. Likewise, applications querying across keysets need the appropriate +term per keyset; no cross-keyset comparison is implied. + +### EQL targets: a concrete adapter boundary + +The example above is a generic searchable layout, **not an EQL payload**. +For EQL, use the adapter's types, metadata, serialization, and query target. +For example, today's `eql_bindings::v4::text::TextEq` already represents an +email ciphertext plus equality term; `TextEqQuery` represents the query. + +The following separate field-level program uses existing APIs and avoids +the proposed `Ciphertext` and passthrough attributes. It needs `eql-bindings` +with its `stack-encrypt` feature, `stack-encrypt`, `serde_json`, `tempfile`, +and `tokio`. It is marked `no_run` because it needs live credentials; it has +been checked against source signatures here, not compiled or executed. + +```rust +use eql_bindings::{ + v4::text::{TextEq, TextEqQuery}, + Identifier, +}; +use stack_encrypt::{DecryptFrom, EncryptInto, StackCipher}; +use std::error::Error; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let column = Identifier::for_column("users", "email")?; + let email = "alice@example.com".to_owned(); + + let encrypted: TextEq = email + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("email.eql.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let query: TextEqQuery = email + .encrypt_into_with_context(&query_keyset, column.clone()) + .await?; + let loaded: TextEq = serde_json::from_slice(&std::fs::read(&path)?)?; + assert_eq!(loaded.hm, query.hm); + + // Supply the expected column, so stored identifier metadata is checked too. + let restored: String = loaded.decrypt_into(&reader, column).await?; + assert_eq!(restored, email); + Ok(()) +} +``` + +`TextEq` includes its EQL version, identifier, ciphertext, and equality term. +Its context is `NonEmpty`. The typed identifier comes from the +application's expected table/column on both writes and reads; decryption +checks it against the stored identifier before opening the ciphertext. +The `v4` namespace is intentional: this Rust producer profile targets EQL +v4, not the TypeScript EQL v3 integrations. + +There is a gap before we can simply replace `SearchableEmail` with `TextEq` +in the outer derive. That derive currently passes `CallerContext`; EQL's +target needs `NonEmpty` and carries its identifier in the output. +The context adaptation must be designed explicitly, including whether caller +extensions are representable. The EQL example demonstrates what that adapter +must preserve without inventing an implicit conversion or a nonexistent SQL +client API. PostgreSQL DDL, parameter binding, and index engagement are a +separate adapter example to add once that boundary is settled. + +## What to review across the three workflows + +| Question | Concrete choice made by these examples | +| --- | --- | +| What is one encrypted unit? | One `OpaqueBlob` encoding in example 1; one selected field in examples 2 and 3 | +| How is the layout selected? | Ordinary encrypt returns an opaque value; `EncryptInto` selects a declared output | +| How is persistence exposed? | Canonical bytes for one ciphertext; Serde composed over a typed layout | +| How is plaintext recovered? | Decrypt/decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | +| How do indexes fit? | Additional field targets under the same layout/execution model | +| Who chooses context? | The application for a blob; the layout plus optional caller extensions for fields; an explicit identifier for EQL | +| Do examples need plans or visitors? | No; those remain implementation and advanced composition mechanisms | +| Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | + +The principal unresolved choices exposed by the examples are the opaque +ciphertext name and storage contract, the public decrypt spelling, the blob +codec and timing contract, EQL context adaptation, and optional source-side output +generation. None requires reintroducing public `encrypt_as`. + +## Source checks and validation limits + +The draft was checked against the existing +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[context composition](../../packages/stack-encrypt/src/target/context.rs), +[derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs), +[decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs), +[canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs), and +[EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). + +All four programs were parsed successfully by `rustfmt` using Rust edition +2024. All local links resolve; code fences and whitespace were checked. +These checks establish syntax and document integrity, not type correctness. + +The candidate APIs are not implemented by this document. The three proposed +programs cannot yet be typechecked against the repository. No live encryption, +database execution, or cryptographic compatibility tests are claimed. Once the surface is agreed +and implemented, promote these programs to executable examples and verify +storage round trips, matching queries, and context mismatches. From 356ba45eee48c8b2032a0360b002fcf89c8cfe36 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 15:35:34 +1100 Subject: [PATCH 2/4] docs: develop Stack Encrypt API design plan --- .../2026-10-24-stack-encrypt-api-plan.md | 1528 +++++++++++++++++ .../2026-10-24-stack-encrypt-api-review.md | 453 ++++- ...026-10-24-stack-encrypt-public-examples.md | 55 +- ...026-10-25-stack-encrypt-json-operations.md | 570 ++++++ .../2026-10-25-stack-encrypt-packaging.md | 253 +++ ...-25-stack-encrypt-provider-capabilities.md | 342 ++++ 6 files changed, 3185 insertions(+), 16 deletions(-) create mode 100644 docs/plans/2026-10-24-stack-encrypt-api-plan.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-json-operations.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-packaging.md create mode 100644 docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md new file mode 100644 index 000000000..600b52091 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -0,0 +1,1528 @@ +# Stack Encrypt API implementation plan + +Date: 2026-10-24. Status: proposed implementation direction, developed from +the [review log](2026-10-24-stack-encrypt-api-review.md), +[public examples](2026-10-24-stack-encrypt-public-examples.md), and +[context examples](2026-10-24-stack-encrypt-context-examples.md). + +This document turns the review into implementation choices and completion +criteria. User requirements are distinguished from recommendations that +have not yet been accepted. Examples below are proposed APIs, not working +code for the current release. + +**Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets +are accepted. The comprehensive JSON first-release contract and Rust/dynamic +examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), +including extraction and walks. ORE does not provide equality initially; +retain a future explicit capability for equality-preserving encodings. +Prioritize stable blob storage bytes via +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); +zeroization/timing implementation improvements may follow without changing +stored format. The naming/package review is now in progress. These decisions supersede +earlier sequencing recommendations below where they conflict. + +**Follow-up decisions:** custom Rust query stages may be explicitly +Rust-only, with `NotPortable` on dynamic export until registered; built-in +capabilities retain Rust/Go parity. Every validation independent of keys +runs before key resolution/loading. The user is still reviewing the JSON +operation contract; its detailed API proposals are not collectively +approved. New keyset-routing, lock-context and provider-capability questions +are assessed in [provider capabilities](2026-10-25-stack-encrypt-provider-capabilities.md). + +**Packaging review:** the user favors retiring Vitamin C `AsyncCipher` and +moving application encryption behavior into Stack. The +[packaging and naming recommendation](2026-10-25-stack-encrypt-packaging.md) +defines crate ownership, an acyclic dependency graph, the proposed shared +format crate, role-based public names and coordinated migration work. It is +a proposal for review, not an accepted rename or an implementation. + +**Compatibility scope (user clarification, 2026-10-25):** this API has only +existed as an early prototype. Do not preserve its APIs, stored ciphertexts, +or encodings for backward compatibility. No legacy decoders, compatibility +aliases, deprecation period, or data migration is required. Choose the +coherent new design and update its Rust, Go, guest, and EQL consumers +together. Engine output must still agree with the SQL that consumes it; +that is current interoperability, not a legacy-format requirement. + +## Requirements established in the review + +- Ordinary encryption is keyset-directed, with `encrypt` and + `encrypt_with_context`. Keep the separate target-directed `EncryptInto` + conversion interface; remove redundant public `encrypt_as`. +- Single-value encryption produces opaque ciphertext; `OpaqueBlob` + explicitly encodes a composite value before sealing it as one value. + Direct `encrypt` does not automatically lift over collections. Use + `EncryptInto` for collection batching, including `Vec` into + `Vec` without caller context. This division is agreed as of + 2026-10-25. The current byte-vector recommendation follows that same + collection rule and uses `OpaqueBlob` for a whole vector; there is no + prototype-compatibility reason to add a raw-byte exception. +- Rust traits and derives are the primary static interface. Plans support + those declarations and dynamic/FFI lowering. +- Design and deliver static and dynamic support together. Go support is an + acceptance criterion, not a later follow-up. +- Provide decrypt entry points with and without explicit context. The + no-context variant must not invent a default required application value. +- `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation + capabilities. Both `encrypt` and `encrypt_into`, including their + context-taking variants, consume input and take responsibility for + cleaning up owned plaintext and plaintext artifacts. Ownership alone is + not a zeroization implementation; require `Zeroize` on plaintext types + and enforce cleanup through guarded storage and lifecycle checks. Deliberate passthrough + fields transfer to the output rather than being erased. +- Keep EQL's engine/SQL contract coherent and retain shared batching/execution + behavior; prototype representations need not be preserved. +- Investigate and replace the EQL JSON adapter's serialization round trips + with direct typed conversion, maintaining engine/SQL agreement. + +## Recommended decision: general configured targets + +Use a general target-plus-typed-options contract. Do not add a public +`json_tree(...)` derive construct. Rust constructs concrete options directly; +the dynamic interface deserializes its options into that same type. JSON +supplies the options type, validation, and implementation behind the general +mechanism. Sharing a contract does not require sharing input syntax or +making static Rust configuration pass through deserialization. + +Three approaches were considered: + +| Approach | Benefit | Limitation | Recommendation | +| --- | --- | --- | --- | +| Typed Rust options factory | Compiler checks the options type; normal Rust APIs and tooling | The function itself is not portable, but its concrete output has a data form | Preferred static surface | +| Named policy type holding options | Reusable and discoverable | Adds a provider trait and marker type where a function can suffice | Optional later convenience | +| Target plus data options | Expressible in Go declarations and dynamic data | Requires deserialization and semantic validation | Dynamic surface, producing the same typed options | + +This supersedes the initial recommendation to put option field literals in +Rust attributes. Rejecting a Rust factory because its code cannot cross FFI +was unnecessarily restrictive: the common boundary is its typed result, +not the code used to construct it. No arbitrary encryption callback is +introduced; the factory only supplies configuration. + +The target owns its source capabilities, canonical output, options/defaults, +validation, encryption description, index/query capabilities, and opening +behavior. Context remains a separate input. Configuration does not supply +arbitrary execution callbacks or let a caller redefine EQL's wire fields. + +Implement configuration as a target capability (working name: +`ConfigurableTarget`), lowering into the existing operation descriptions. +Its associated options type has a data representation. Rust factories +return that type directly; registered dynamic targets deserialize into it. +Validate and normalize once when a reusable declaration is built. Both +paths use the same target-owned semantic validator, before key I/O. Use +validated deserialization, for example a wire helper plus `TryFrom`, rather +than a derived deserializer that bypasses checked builders. Keep any raw +options fields private if the public type promises validated invariants. + +Compile-time guarantees cover factory resolution and signatures, the +target/options association, and typed builder arguments. They do not imply +that a string is a valid JSON path or that the factory evaluates at compile +time. JSON path construction is currently fallible. Begin with a fallible +factory contract, `fn() -> Result`, with errors converted +to the engine's configuration error path. Propagate errors without panics +or `unwrap`; a later literal-path macro can add stronger static validation. + +Resolve the factory once per reusable configured declaration and reuse its +normalized options for writes and queries. A factory can read mutable +external state, so its type alone cannot prove a fixed policy; document +that fixed-policy providers must be stable. Do not independently reevaluate +the provider when generating a query. Intentionally variable configuration +belongs in an explicitly constructed configured-target instance. + +Unlike today's `Layout`, the general capability must not require one index: +targets may have none or multiple. Adapt the existing JSON layout internally +while migrating; do not expose two competing configured-target concepts. +The target catalog resolves registered targets for Go. Custom Rust targets +remain possible without registration, but cannot be dynamically addressed +until their adapter and descriptor are included in the guest build. + +## Candidate static syntax + +`target(options = payroll_options)` names a Rust factory for the inner +target of a transparent newtype. The inner Rust type selects the output +representation and associated options type. The spelling is proposed; +typed factories for Rust and typed deserialization for dynamic callers are +the revised recommendation. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use stack_encrypt::json::JsonOptions; +use stack_encrypt::sem::TermError; + +fn payroll_options() -> Result { + JsonOptions::new().ordering(["$.salary"]) +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = serde_json::Value, + transparent, + target(options = payroll_options) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); +``` + +The factory body uses today's `JsonOptions` builder; the derive support is +proposed. The derive calls the factory and type-checks its result against +the inner target's associated options type. It knows neither JSON path +semantics nor JSON-specific option names, and does not serialize or +deserialize the factory's result. The configured target constructs its +output, which the derive wraps. Serde transparency preserves that output's +wire envelope; SQL driver traits are a separate integration requirement. +The same factory attribute should work on a field of a derived output +record, with existing context selection rules. + +```rust +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored = serde_json::Value::decrypt_from_with_context( + encrypted, &keyset, column, +).await?; +``` + +Fixed policy is a recipe for writes and queries. Transparent deserialization +does not attest that an existing row was created with that policy. In +particular, the full ordered-path policy is not stored in the JSON header. + +## Candidate dynamic representation and Go requirements + +Extend the existing string target with an object form, using the current +type name provisionally rather than deciding a registry rename here: + +```json +{ + "payroll": { + "context": ["employees", "payroll"], + "target": { + "name": "JsonTreeDocument", + "options": { "ordered": ["$.salary"] } + } + } +} +``` + +Keep `"target": "TextEq"` as shorthand for that target's default options. +The object form is new: current dynamic targets accept only a string. +Reject unknown options, incompatible options, unsupported target names, and +invalid paths during declaration validation. Do not silently fall back to +defaults. Specify one consistent JSON wire format and context encoding +across the Rust, dynamic, Go, and database paths. + +Go's generator and policy API must represent these same options, emit the +same declaration, and obtain the EQL value from the guest. Go must not +assemble the EQL envelope or reproduce JSON indexing. Discovery must expose +the options contract and capabilities; extend the existing target catalog +and resolver rather than installing a parallel JSON registry. +The dynamic registry invokes the concrete target options deserializer; it +does not look up or invoke the Rust application's factory by name. A Go +caller supplies equivalent options data. If a Rust policy must be exported, +serialize its resolved options, not its function identifier. This contract +does not require all Rust-only targets to depend on Serde; data conversion +is required for targets registered with the dynamic interface. + +Current gaps: JSON tree is absent from EQL's catalog/target table, the +dynamic JSON index is refused with `JsonIndexPending`, target fields carry +only a name, and Go's generated target declarations carry no target options. +Supporting an index's option syntax in Go does not establish executable +JSON-tree support. Include JSON input transport for objects, arrays, scalar +roots, and null; preserve the engine's numeric semantics and reject values +the cross-language contract cannot faithfully represent. + +Writes and queries must reuse the normalized configured target in Rust and +Go. An ordering query on an undeclared path must fail before key I/O. +Containment, path equality, ordering, and extraction need capability metadata +that can express their different operand/result shapes, not a single +assumed scalar query twin. Decryption must use stored cryptographic metadata +and context, not require the caller to remember the write's ordered paths. + +## Accepted decrypt surface and return ownership + +**Decision, 2026-10-25:** retain ordinary keyset-directed decryption for one +opaque ciphertext and destination-directed `DecryptFrom` for declared +targets/layouts. Both paths have context-free and context-taking variants: + +```rust +let value: i32 = keyset.decrypt(encrypted).await?; +let value: i32 = keyset.decrypt_with_context(encrypted, context).await?; + +let plaintext = User::decrypt_from(encrypted, &keyset).await?; +let plaintext = User::decrypt_from_with_context(encrypted, &keyset, context) + .await?; +``` + +These are independent call examples, not sequential reuse of a consumed +ciphertext. Direct decrypt mirrors direct encrypt; `DecryptFrom` mirrors +target-directed `EncryptInto`. The existing input capabilities still govern +whether omission of context is available. Do not add a competing public +ciphertext-receiver `decrypt_into` method. + +Successful decryption returns an ordinary owned `T: Zeroize`. Stack guards +recovered bytes and intermediate plaintext while it owns them; successful +return transfers responsibility for the value to the caller. `Zeroize` does +not automatically wipe that returned value on drop. Support explicitly +requested drop-guarded results without requiring a wrapper around every +decrypted application struct; the precise opt-in adapter is implementation +design within this decision. Guarding the completed value does not resolve +partial-deserialization cleanup, which remains required work. + +This reads as constructing `User` from ciphertext. Calling +`encrypted.decrypt_from(...)` would conflict with the ordinary From/Into +receiver convention; retain the agreed destination receiver. +The destination-directed pair already exists on the blanket `DecryptFrom` +extension trait. Today's ciphertext method is +`encrypted.decrypt_into(cipher, context)`, also on `DecryptFrom`, and has +no separate context-free overload. The current declaration trait +`DecryptInto

` is another layer; do not rename it accidentally while +settling method ergonomics. Consolidate declaration and convenience-trait +naming with review issue 5. The public examples use the destination-directed +pair; remove the competing ciphertext method when implementing this surface. + +## Recommended public traits and context input capabilities + +Status: direction accepted by the user on 2026-10-24: separate conversion +methods from `EncryptTarget`/`DecryptSource` declarations, and use explicit +caller-context capabilities instead of `Default`. Implementation, complete +signatures, and call-site updates remain pending. This section resolves the +direction of work areas A and B below; earlier examples retain historical +derive names until their migration. + +Keep two user-facing conversion traits, implemented by blanket library +implementations: + +| Trait | Public methods | Receiver | +| --- | --- | --- | +| `EncryptInto` | `encrypt_into`, `encrypt_into_with_context` | Plaintext value; output inferred or selected | +| `DecryptFrom` | `decrypt_from`, `decrypt_from_with_context` | Plaintext destination type; ciphertext supplied as an argument | + +These methods run declarations through the common engine and return the +existing batched awaitable. Target implementers do not override execution. +Ordinary `keyset.encrypt` and `encrypt_with_context` remain inherent methods +and do not gain a plaintext-side `encrypt` synonym. Their accepted decrypt +counterparts are `keyset.decrypt` and `decrypt_with_context`. + +Give the declaration traits role names, rather than From/Into names that +promise conversion methods they do not define: + +| Proposed declaration | Implemented on | Describes | +| --- | --- | --- | +| `target::EncryptTarget

` | Encrypted output or term target | How to produce this target from `P` | +| `target::DecryptSource

` | Recoverable encrypted output | How this stored value yields `P` | + +The corresponding derives should use the same names. This replaces today's +declaration traits `EncryptFrom

` and `DecryptInto

`; it does not remove +the retained `EncryptInto` extension trait. A term-only target implements +only `EncryptTarget`: no decrypt method that always fails. Plaintext visitor +traits remain a separate Stack-owned representation layer; moving/renaming +them must retain a single canonical trait implementation and a consistent +wire contract across the new consumers; prototype compatibility is not needed. +Configured-target options are another associated capability, not context. + +Declaration methods continue returning core-owned operation descriptions. +Encryption declarations do not receive the plaintext and cipher together; +decrypt declarations inspect stored output and select its opening, without +performing key I/O. Plans, derives, and dynamic resolution reuse those +descriptions. Keep the declaration module public for custom targets and +keep normal method imports/derives available from the Stack entry point. + +Use `ContextInput` as the associated-type name on each declaration trait. +It describes only what the caller must or may supply. It is not the complete +cryptographic context, which can also include a prefix and fields selected +from plaintext or stored output. Encrypt and decrypt declare this type +independently. + +Recommend library-owned input wrappers with explicit capabilities: + +| Proposed input | Omitted argument | Supplied argument | Meaning | +| --- | --- | --- | --- | +| `NoCallerContext` | Allowed | Not accepted | Declaration derives all context and exposes no caller input | +| `RequiredContext` | Unavailable | Required `C` | Supplies context or the missing component selected by the declaration | +| `OptionalContext` | Allowed | Accepted `C` | For ordinary ciphertext: omission means empty context; supplied context is used directly | +| `ContextExtension` | Allowed | Accepted `C` | Omission preserves the declared base; supply extends it | +| `ExpectedContext` | Allowed | Accepted `C` | Omission reconstructs stored context; supply additionally checks it | + +`OptionalContext` does not append a synthetic empty prefix, and omission +of an extension does not append `()`. `ExpectedContext` never extends AAD. +The table specifies behavior; names may be refined without changing it. +Nonempty requirements belong to the accepted argument type or validated +resolved context, not to whether an input is optional. Search targets still +require a nonempty effective context. Ordinary scalar ciphertext should +accept the same empty context through both keyset and target-directed APIs. + +No-context methods require an explicit, sealed `AllowsNoContext` capability; +`_with_context` methods require a corresponding `AcceptsContext` capability +with its accepted argument type. Neither is inferred from `C: Default`. +`RequiredContext` consequently remains required even though `u64` +defaults to zero. The helpers accept ordinary argument values (or ergonomic +checked conversions); users do not manually wrap `RequiredContext` or +`ExpectedContext` at every call. Conversion errors propagate before key I/O. + +The core-owned wrappers carry presence and intent; target declarations +perform source selection and context composition. For composite required +input, `C` may be a tuple or named struct; it does not have to be the full +logical context. Preserve existing grouping rules and context encodings. + +Examples of independent encrypt/decrypt requirements: + +| Declaration | Encrypt input | Decrypt input | +| --- | --- | --- | +| Caller row ID stored in output | `RequiredContext` | `ExpectedContext` | +| Source row ID also stored in output | `NoCallerContext` | `ExpectedContext` | +| Source row ID not recoverable before decrypt | `NoCallerContext` | `RequiredContext` | +| Static field base with optional unstored extension | `ContextExtension` | `ContextExtension` | + +In the last row, decrypting an extended ciphertext requires supplying the +same extension; availability of the omitted method is not a promise that +it opens every ciphertext of that Rust type. A derive can expose an optional +extension instead of `NoCallerContext` where its declaration explicitly +allows that behavior. Expected-context checks do not authenticate stored +data themselves: successful cryptographic opening remains necessary. + +The dynamic plan must encode the same requirements, source selectors, and +storage rules. Rust enforces omitted/supplied method availability with trait +bounds; the engine enforces dynamic input presence and validity before key +I/O. Omitted input, explicit zero, and an empty supplied value are distinct. +Go generation can expose convenient signatures using that metadata, but +the guest remains authoritative for runtime requests. + +Implementation proof points: compile-fail required-input calls even for +`C: Default`, reject context for a no-input-only target, preserve optional +extension versus expectation semantics, and execute all nine directional +scenarios through Rust and dynamic plans. Preserve owned-source support +and batching; this trait separation must not add a blanket `Clone` bound. +The final signatures must show where borrowed and owned sources are +supported rather than concealing the existing difference behind the rename. + +## Parallel Go JSON work: PR #1353 + +Read on 2026-10-24: [PR #1353](https://github.com/cipherstash/stack/pull/1353) +is an open documentation-only plan, not an implementation. Its write-up is +[The json_tree capability in the Go SDK](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md). +It tracks [#1347](https://github.com/cipherstash/stack/issues/1347), with +catalog, dynamic execution, queries, Go SDK, and extraction work in +[#1348](https://github.com/cipherstash/stack/issues/1348), +[#1349](https://github.com/cipherstash/stack/issues/1349), +[#1350](https://github.com/cipherstash/stack/issues/1350), +[#1351](https://github.com/cipherstash/stack/issues/1351), and +[#1352](https://github.com/cipherstash/stack/issues/1352). + +Treat that work as an implementation dependency to reconcile, not a parallel +SDK design to recreate. Its engine-owned execution, catalog/generated Go +types, engine-validated options, and cross-language tests fit this plan. +Review the following differences before freezing interfaces: + +- Route its JSON-specific option plumbing through the general target's + typed options/deserialization boundary. Go tags may retain idiomatic + syntax, but lower to target options rather than a second JSON policy path. +- Adapt the proposed `Layout` integration to the configured-target contract; + no second executor or permanently competing public configuration model. +- Reconcile type/registry names and generated query accessors with the shared + glossary and configured query capabilities. Its `At.*` accessor proposal + is useful input, not an accepted Rust or Go naming decision here. +- Carry the agreed context requirements and paired method semantics into + the relevant language interfaces without requiring identical language + syntax. Distinguish JSON `null` as content from nullable SQL fields. +- Evaluate its JSON-text ABI recommendation for numeric fidelity and return + types. This is compatible with direct typed EQL output adaptation: input + transport text and an avoidable encrypted-output Serde round trip are + different boundaries. +- Release scope is now decided: extraction, opening entries and walks ship + in the comprehensive first version alongside writes/queries. The earlier + Go plan's step-5 deferral is superseded by the linked JSON operation + inventory; sequential internal PRs must not become a reduced release. +- Byte-for-byte parity tests need deterministic randomness as that plan + proposes; normal randomized encryption needs cross-opening/query tests. + +These are dependencies and recommendations recorded locally. No changes or +comments have been posted to the parallel PR or its issues. + +## Prior query design: baseline and reconciliation + +**Decision update, 2026-10-25:** ORE does not answer equality in the initial +API, for any plaintext type. Equality uses its declared equality-capable +index. Keep capability selection parameterized by the index, plaintext +encoding/domain and options, so a future explicit exact/injective integer +encoding can opt into equality without changing query syntax or wire +operator identity. Never infer `Eq` merely from ordering support: text and +other lossy encodings cannot establish plaintext equality. A future opt-in +must retain overlap rejection (one selected provider per operator) and +must not silently invalidate existing `(Equality, Ore)` definitions. This +policy does not disable JSON scalar equality, which uses value selectors +rather than ORE/OPE terms. + +Read in full on 2026-10-25: the user-supplied **Stack Encrypt query design**, +dated 2026-10-21, at +`/Users/dan/Downloads/Stack Encrypt query design.md` (654 lines). The source +document was not modified. It is explicitly a discussion draft, based on +an earlier `next` snapshot; its historical code/release claims were not +revalidated as part of this reading. Use it as the query-design baseline, +not as a settled specification or a reason to redesign queries from scratch. + +### Existing proposal to carry forward + +- The caller selects an operator and a typed field/target handle. The field + supplies its own configured index; remove `DefaultIndex` and caller `.via`. +- Each field has at most one index answering each operator. Refuse overlaps + at declaration validation instead of choosing an index by priority. +- Declare operators, index capabilities, and admitted source types once; + derive or verify typed lookup, dynamic lookup, and discovery from that + declaration. Avoid another capability table in EQL. +- Share query derivation and refusals through the same semantic functions. + In particular, tokenize match input once and retain the tokens needed for + probabilistic verification in the query object. +- Retain index/source types in typed field handles. Unsupported operators + are compile errors on the typed path and validated errors dynamically. +- Model analyzer case behavior once in its stages, rather than restating + it in both the match configuration and the operator. + +The draft already proposes the query call +`cipher.query(op::Matches, text).on(&field).await?`, typed +`FieldRef` handles, and a loaded configuration/analyzer owned by the dynamic +plan (`Loaded` / `Arc`). Those are substantive designs to reuse; +they do not yet settle how our output-derived typed factory options are +initialized and shared with convenient `encrypt_into` calls. + +### Reconciliation with this review + +1. **Generate handles from the same output-target declaration.** The query + appendix proposes a plaintext-side `EncryptionPlan` derive and generated + encrypted counterparts. Our agreed starting point is output-side + `EncryptTarget`/`DecryptSource` derives. Recommend emitting typed handles + from that declaration, without separately restating indexes on plaintext. + A source-side convenience derive can later generate the same target. + Hand-written plans remain lower-level composition, not mandatory usage. +2. **Resolve typed factory ownership.** A generic `Match` can fix a policy + identity in its type, but an arbitrary factory can still compute different + values on different calls. Carry a resolved configuration instance through + writes and field handles; settle the owner/initialization API. The draft's + dynamic plan ownership is useful precedent, not proof that all static + factory results are compile-time constants. +3. **Separate portable options from execution types.** JSON's Rust factory + and dynamic deserializer can construct the same `JsonOptions`. The draft's + `Match` and `Match` intentionally have different execution + types to accommodate generic/custom analyzer stages. Preserve shared + validated options and semantic implementations for the portable subset; + do not force all typed analyzers through a serializable stage enum. +4. **Accepted portability scope.** Ship built-in advertised + capabilities in Rust and Go together. Downstream custom Rust stages can + remain explicitly typed-only with `NotPortable` until registered. This + qualifies the draft's blanket "lands in Rust first" rule; it must not + become permission to defer the Go JSON capability under this plan. +5. **Include JSON capabilities before fixing lookup representations.** The + draft's 16-shape dynamic enum covers equality, match, and scalar ordering, + not the new JSON target. Extend operator identity, operand/result shapes, + capability validation, and overlap rules for containment, `AtPath`, and + extraction. Its interim validated index list using one capability table + is a viable starting point; the specific closed enum is not a decision. +6. **Use the agreed context/scope model.** Replace label-only assumptions + with separate ciphertext binding and search scope. Row binding does not + flow into search terms; `PrimaryKey` is context-eligible and `ScopeKey` + is also scope-eligible. Apply the same declared scope to index generation + and queries. Preserve the requirement to return or supply row context + when decrypting selected results. +7. **Accepted validation timing.** All checks independent of keys precede + key resolution/loading, including query-input checks. The draft's possible + named-keyset resolution before those checks is superseded. Reuse the + same validator for Go's `check_query` and execution rather than copying + it or tokenization. Provider facts and cryptographic checks that require + keys remain later stages. Explicit earlier client initialization/keyset + calls are not retroactively prevented by this operation-validation rule. +8. **Apply the user's prototype scope.** Legacy match-option decoding, + deprecation/release-preservation requirements, and frozen Go wire formats + in the older document are not constraints here. Keep new Rust/Go/EQL + consumers aligned. The appendix's `cipher.encrypt(plan, value)` and + plaintext derive are illustrative older proposals, not replacements for + our agreed entry points. + +ORE/Eq is now decided as described above. Retain integration work on +typed field-handle construction/spelling, dynamic +`Index` versus a shared semantic-function boundary, feasibility +of compile-time match-option checks, Go's `check_query`, and ADR updates. +Type-safe stage ordering does not by itself validate runtime numeric values +or string paths; stronger const-check claims require implementation proof. +Use these as specific integration decisions rather than an unbounded query +API task. Coordinate their Go surface with PR #1353. + +## Recommended owner: a reusable typed target definition + +Status: recommendation for review. Use an immutable **target definition** +to own resolved options, prepared indexes/analyzers, context-selection rules, +and operation metadata. The output derive generates this definition's typed +field accessors from the same declaration that generates encryption. It +contains no plaintext, row values, credentials, or selected keyset. A field +handle borrows or shares that definition and retains its plaintext/index +types; it is not an independent name plus a newly constructed default index. + +**Selector naming, 2026-10-25:** the user finds encryption's `using` and +query's `on` confusing and requests considering one name. Recommend +`.using(...)` for both: each selects the definition or field whose declared +policy governs the operation. This spelling is proposed, not yet accepted. +Encryption still accepts the requested target's associated definition type; +query still requires `QueryTarget` on a selected field or +single-value definition. Shared naming does not weaken either bound or +make an entire multi-field record a query target. Older `on` examples in +the supplied query draft describe its earlier spelling. + +Proposed usage, assuming the previously described `EncryptedUser` target: + +```rust +let users = EncryptedUser::definition()?; + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + +let query = cipher + .using(&users.email()) + .query(op::Eq, "alice@example.com") + .keyset(&keyset) + .await?; +``` + +`definition()` obtains the shared resolved definition of that source/target +mapping. Plain `encrypt_into` obtains the same definition internally. It +does not independently invoke the options factory. `.email()` is generated +from the output field declaration, not a hand-maintained second schema. +The method name `definition` is proposed; it names a resolved declaration, +not a builder that users must reconstruct before every operation. + +### Call order and strict typestate + +User requirement, 2026-10-25: use strict typestate for the fluent API. The +user prefers the reading `cipher.using(&field).query(op, input)` and is open +to supporting both call orders. Recommend definition-first as the canonical +documented query form, while supporting operator-first through the same +request states: + +```rust +let a = cipher.using(&users.email()).query(op::Eq, email).await?; +let b = cipher.query(op::Eq, email).using(&users.email()).await?; +``` + +Both are proposed. The current implementation is query-first, followed by +`.on(...)`; this recommendation changes that interface. + +| Starting state | Method | Resulting state | +| --- | --- | --- | +| Cipher | `using(&definition)` | Bound definition, no query selected | +| Bound definition | `query(op, input)` | Query with definition/operator/input known | +| Cipher | `query(op, input)` | Query awaiting a definition | +| Query awaiting a definition | `using(&definition)` | The same query state as the definition-first path | + +The transition that brings the definition and operator/input together has +the `QueryTarget` bound. A definition-first bound handle need not +pretend to know the future operator. Once combined, the index and context +requirements are determined. Only the executable state implements +`IntoFuture`; a bound definition alone, a query missing its definition, or +a query missing required context cannot be awaited. Required context is +completed by a typed context-input transition; optional/no-input cases +start with a valid omitted state. Context completion must not be represented +solely as an unchecked `Option` plus a runtime missing-input error on the +static path. + +Do not expose another `using` transition after a query's definition has +been selected, or another query-selection transition on the request. +Supporting these two orders does not promise every permutation of keyset, +context, and operation selection. Specify those transitions separately, +starting with context supplied after its required type is known. Share +the final query request, validation, and execution implementation rather +than duplicate them per call order. Input-dependent errors such as an +invalid path still need runtime validation; typestate checks completeness +and capability, not arbitrary value semantics. + +The same discipline applies to encryption requests: a fixed-policy +`encrypt_into` has an implicit definition and can be awaited without +`using`; selecting an explicit compatible definition is a one-time +transition before execution. This does not add a new ordinary `encrypt` +receiver or require fixed-policy callers to bind definitions manually. + +Completion tests must include both query orders with identical semantics, +and compile-fail cases for unsupported operators/inputs, premature await, +missing required context, and repeated definition selection. Definition +borrows/shared handles must remain valid for the lifetime of the request. + +### Fixed declarations + +Use lazy, fallible initialization for the fixed definition, retaining each +successful result for reuse. Allow explicit startup initialization via +`EncryptedUser::definition()?`, while the shorthand write initializes it +on demand and returns any error through its awaitable. Factory errors must +not panic or perform key I/O; do not permanently cache initialization +failures. Coordinate concurrent first use so successful initialization is +performed once, and account for reentrant/recursive initialization in the +implementation rather than allowing a deadlock. + +The identity is the complete concrete declaration (including source/target +mapping and generic policy parameters), not merely the ciphertext's output +type. Different instantiations must not share a cache slot by accident. +This is an implementation requirement, not a claim that a static inside a +generic Rust function is automatically per-monomorphization. + +Fixed factories must describe stable application policy; they are not the +interface for reloading environment settings or tenant-specific options. +The cache makes writes and queries in one process share the resolved policy; +it does not prove another process or an older stored row uses that policy. +Per-call context, source values, and keyset choice remain separate inputs. + +### Runtime declarations + +Load/create a distinct definition instance for runtime options. A configured +instance is not stored in the fixed-declaration cache. Use the same type of +definition and field handles, so queries and writes share that instance. + +Proposed Rust spelling for a configurable library target: + +```rust +let payroll = EqlEncryptedJsonTree::configure(options)?; + +let encrypted: EqlEncryptedJsonTree = value + .encrypt_into_with_context(&keyset, column) + .using(&payroll) + .await?; +``` + +Here `using` takes a resolved typed target definition, not a hand-built +plan. This requires a lazy conversion request: select the definition before +building the operation or invoking a default factory. Its await lowers to +the existing `Pending`/batch executor; the shared engine remains unchanged. +The definition's output/source types constrain the conversion. Queries use +the same definition (or its field handles), supplying the required context +and keyset for that call. + +A fixed-policy wrapper such as `EncryptedPayroll` does not expose runtime +override of its declared options. Use the configurable underlying target +for the runtime case, avoiding silent overrides of a fixed policy. Exact +configuration APIs for multi-field runtime definitions are still to be +specified; they must select typed options without introducing another +execution mechanism. + +Go's loaded dynamic plan owns its resolved definitions and generated field +accessors reach those definitions. It has no Rust factory to invoke. The +guest validates and builds the configuration once per loaded declaration, +then writes/queries use the loaded instance. A different runtime policy is +a different instance; it does not mutate one used by in-flight operations. + +### Implementation proof and boundary + +Prove the cache/handle design early with two fixed mappings, multiple generic +instantiations, concurrent first use, initialization failure/retry, and +separate runtime instances. Prove a query and its write use the same options +without calling the factory twice. Preserve source lifetimes, owned-source +support, batch lowering, and no-HTTP/WASM builds. Queries on a field without +the operator must remain compile errors, with dynamic equivalents checked +at declaration/query validation. + +Prefer this generated-definition approach over requiring callers to author +a second plan or putting configuration state in each ciphertext. Plans are +still the lower-level implementation of the definition. Portable option +serialization and index capability discovery remain part of its dynamic +contract; the definition object does not itself need to serialize arbitrary +Rust closures or custom analyzer code. + +## Single-ciphertext representation and plaintext ownership + +Accepted decisions, 2026-10-25: use `OpaqueEncode` / `OpaqueDecode` for +single-ciphertext representation, replacing the working names `EncodeValue` +/ `DecodeValue`. Both direct `encrypt` and target-directed `encrypt_into` +consume their values and take responsibility for owned plaintext cleanup, +including their context-taking variants. These are representation +capabilities; the engine still performs cryptography. `EncryptTarget` / +`DecryptSource` describe how encrypted outputs and layouts compose those +operations. The lifecycle and cleanup scope below are part of the accepted +design. The plaintext cleanup bound is `Zeroize`; implementation remains +to be completed. + +### Owned input is the direct API contract + +`keyset.encrypt(value)` and its context-taking counterpart take an owned +single-value input. No implicit clone and no `Clone` bound. For example, +passing a `String` transfers it; a borrowed `&str` does not transfer the +storage it refers to and must not receive the same cleanup promise. Do not +add a silent borrowed-input fallback to the consuming operation. + +**Decision, 2026-10-25:** require `Zeroize` for plaintext types and use a +library-owned guard. Use the existing ecosystem trait rather than defining +a new cleanup trait or alternative protected-ownership adapter for now. +`Zeroize` supplies the operation; the library must arrange to invoke it on +every supported exit path. A bound alone does not arrange cleanup on drop. + +Acquire protected ownership synchronously when creating the encryption +request, before returning its future/builder. An `async fn` body is not run +until polling, so placing the first guard inside that body would leave an +unpolled request with an ordinarily dropped source. A sync constructor +returning the typed future can establish the guard first. Every subsequent +builder transition moves that guarded ownership rather than making a new +unprotected copy. + +The representation trait may encode through a temporary internal borrow of +the guarded value into a library-owned sensitive sink. Public consumption +does not require giving an encoder unrestricted ownership of the source or +an ordinary growable `Vec` for scratch work. Wipe and release the source +as soon as encoding no longer needs it; keep encoded plaintext guarded until +sealing completes. Preserve the single-leaf representation boundary. + +### Cleanup scope and implementation obligations + +- Cover owned source storage, encoded buffers, codec scratch, and any + library-created plaintext copies. Use established non-elidable wiping + primitives/protected storage rather than `clear()` or ordinary drop alone. +- Cover success, encoding/configuration/KMS/AEAD errors, partially prepared + batches, request cancellation, and an unpolled request being dropped. + Use drop guards for ordinary unwinding too; process abort or a leaked + future is not a destructor guarantee. +- Avoid reallocations that abandon uncleared buffers; growing sensitive + sinks must have a cleanup strategy for old allocations. Prevent plaintext + from escaping in diagnostics, Debug, caches, and instrumentation. +- On decrypt, protect recovered bytes and partial decode state until cleanup + or successful ownership transfer to the caller. Successful returned + plaintext is intentionally retained; its subsequent lifetime is the + caller's responsibility unless the return type itself is protected. +- A Rust move does not prove all physical copies were erased. In particular, + `Copy` inputs leave caller-owned originals usable, and compiler temporaries + may exist. The enforceable API promise concerns supported owned storage + and library-managed artifacts, not caller-retained copies or erasure of + every register/stack remnant. Verify stronger claims with compiler evidence. + +The [zeroize documentation](https://docs.rs/zeroize/latest/zeroize/) +documents drop-based guards, non-elidable wiping, and limits involving +moves, stack copies, and reallocations. The source currently uses +`Protected>` at the AEAD boundary, but that does not establish +end-to-end cleanup of every caller value and intermediate. This discussion +is a design requirement, not a completed source/assembly zeroization audit. + +### Consequences for blobs, layouts, and Go + +`OpaqueBlob` cannot promise source cleanup from `T: Serialize` alone. +Require `T: Serialize + Zeroize` for the proposed Serde-backed blob input; +owned blob decoding requires `T: DeserializeOwned + Zeroize`. Ordinary +application structs may derive `Zeroize`. A +generic Serde implementation can allocate or retain hidden copies, so strong +claims also require a supported codec/implementation contract; wrapping its +final byte buffer does not make arbitrary user serialization code compliant. + +In particular, `T: Zeroize` only supplies cleanup once a `T` exists. If +derived `Deserialize` allocates a `String` field and fails on a later field, +the partially built fields ordinarily drop without invoking `T::zeroize`. +Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix a struct +that was never constructed. Keep this as a concrete decoder design and +acceptance requirement; do not claim the new bound solves partial decoding. + +### Blob codec assessment: Vitamin C frames + +**Priority decision, 2026-10-25:** first define a stable, versioned binary +storage profile. The codec was originally ephemeral transport; stored blobs +need a stronger compatibility commitment. Zeroization and constant-time +implementation improvements can follow, provided they preserve stored +format semantics. Do not defer decisions that affect stored bytes (option +presence, type mapping, versions) or claim timing/cleanup properties not +implemented. The `Zeroize` input bound and ownership direction remain. +Track this action in +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519), created +for this review. It explicitly distinguishes storage stability from the +transport-only rationale in Vitamin C #332. + +**Recommendation, 2026-10-25:** reuse Vitamin C's frame writer/parser as the +foundation for the library-owned blob codec, with a specified blob Serde +mapping. Do not adopt the existing transport Serde mapping unchanged. +The inspected dependency is `vitaminc-aead-frame` 0.8.0; its `serde` module +already provides `to_vec(&T)` and `from_slice::(&[u8])`. Stack currently +uses that module for dynamic control data. Its module documentation +explicitly scopes it to transport data rather than plaintext to seal. + +The implementation already guards the serializer output, temporary variant +buffers, and buffered map keys with `Zeroizing`. The underlying +`vitaminc-aead` frame writer grows into a fresh allocation and wipes the old +one. However, `to_vec` returns an ordinary `Vec` on success. Provide an +internal guarded-output entry point or sensitive sink for the blob path, +keeping the result guarded through sealing. Inspect remaining temporary +storage and custom Serde paths before asserting a complete cleanup contract. + +The current mapping is not lossless for every ordinary Serde type: +`serialize_some` delegates directly to its value, while `None` and unit both +encode as null. Thus `Some(())` reads back as `None`; nested optional values +have the same ambiguity. The blob mapping must preserve option presence +explicitly, using an unambiguous representation, and test these cases. +The current map representation also requires text keys. Specify support or +explicit rejection for non-string map keys, and document enum/buffered Serde +limitations rather than advertising all `Serialize` types as round-trippable. + +Frame parsing validates one complete frame before invoking `Deserialize`, +but that neither prevents type-level decode failures nor cleans up partial +application values. Resolve that cleanup boundary alongside format limits, +authenticated codec versioning, and cross-language fixtures. Reusing the +frame format is not a constant-time guarantee and does not make an opaque +blob a structurally encrypted JSON document. + +[Borsh](https://docs.rs/borsh/latest/borsh/) remains a fallback if the blob +mapping would require disproportionate changes. It introduces +`BorshSerialize` / `BorshDeserialize` instead of the Serde traits, and has +writer-based serialization. Changing serialization libraries does not by +itself establish wiping or a constant-time contract. Prefer evaluating the +existing frame machinery against these requirements first. + +The same consuming contract applies to target-directed `encrypt_into`. +For a structural target, declared passthrough fields intentionally +move into the output; wipe encrypted source fields and temporary copies, +not plaintext fields the caller explicitly chose to retain. Fan-out to +ciphertext and index operations should read a guarded source where possible +instead of requiring a plaintext clone per operation. + +Specify cleanup for dynamic host/guest transport and JSON parsing buffers +as part of the Go implementation. The Rust guest cannot erase Go caller +objects it never owned; host-managed copies need host cleanup and ordinary +Go values do not acquire Rust move semantics. Document that boundary rather +than claiming guest cleanup erases every copy in the calling process. + +Acceptance work: source-cleanup/drop observation on successful and failing +requests, cancellation/unpolled-request tests, nested source/blob cases, +and owned non-Clone inputs. Inspect optimized code for wipe/copy behavior +where necessary; ordinary tests alone cannot prove absence of every residual +copy. Do not infer a current vulnerability or a completed guarantee from +this design discussion. + +## Configured-target implementation sequence and completion criteria + +1. **Normalize the general contract.** Specify typed options, Rust factory + signatures, validated dynamic deserialization, defaults, context + inputs, capability discovery, errors, and the target + object grammar. Create the versioned interoperability specification, + schemas and initial conformance fixtures described below alongside this + work. Build one target-owned validator and lowering path. +2. **Deliver JSON as a complete first implementation.** Add configured EQL + JSON targets, direct typed output/input conversion, catalog/guest + registration, and JSON transport. Keep engine and database encodings aligned. +3. **Expose both declarative front ends.** Add the generic Rust derive + factory hook and Go generator/policy support, including field targets and + standalone Rust wrappers. Neither front end reimplements encryption. +4. **Complete queries and reads in both languages.** Carry the configured + target into query construction, expose extraction results, and standardize + Rust's paired decrypt convenience methods. Settle the role-based JSON + names as part of this public API work. +5. **Verify the complete slice before calling it supported.** Rust-written + EQL JSON must open and query from Go, and vice versa, against PostgreSQL. + Test defaults, ordered paths, both root settings, arrays/scalar roots/null, + numeric boundaries, missing paths, context mismatch, and invalid options. + Assert logical batching and verify fixtures for the chosen format. Compare + semantics rather than ciphertext equality, since encryption is randomized. + Run the shared interoperability conformance suite and gate schema/runtime + drift in CI. Each implemented contract change updates its specification + and fixtures in the same change. + +These are implementation steps, not permission to ship a Rust-only phase. +The Go examples and cross-language acceptance tests are part of the feature. +Do not add the plaintext-configuration wrapper in this first change: the +dynamic target declaration already handles runtime options without adding +another policy location. Reconsider that convenience after both front ends +use the shared contract. + +### Required deliverable: dynamic interoperability specification + +**User requirement:** develop a formal specification of the dynamic +interoperability layer during implementation, not as documentation added +after the Rust and Go APIs have diverged. The current examples remain +proposals until reconciled into that specification; adding this deliverable +does not claim the specification has already been written. + +**Recommended format:** a versioned specification package containing +[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) schemas, +normative semantic and wire-format rules, and executable conformance +fixtures. JSON Schema describes declaration/request/response structure; +it is not the entire behavioral or binary transport specification. Keep +application semantic-type declarations distinct from JSON Schema's own +`type`, `$defs`, and `$ref` vocabulary. + +Required coverage: + +- Definition grammar: built-in and named types, optional `types` section, + inline role wrappers, context selectors, scope eligibility, passthrough + and context storage, targets/options, query capabilities, defaults, and + unknown-field/extension handling. Named ordinary types need no role; + naming a type does not grant scope eligibility. +- Semantic validation: resolve named types and field references, reject + invalid/cyclic references where unsupported, enforce primary-key versus + scope-key capabilities recursively, check source/output compatibility, + and derive write/query scope agreement. Do not imply generic JSON Schema + validators perform the engine's reference resolution or capability checks. +- Operation contracts: input presence and context/scope roles, target and + definition identity, results, stable error codes and locations without + plaintext values, validation stages, and which failures precede key I/O. +- Wire mapping: JSON and frame representations, integer widths/ranges and + lossless numeric transport, bytes/text encodings, missing versus null, + context ordering/grouping, descriptor/AAD/index-scope lowering, and + normative references to the ciphertext/SQL envelope and tag extraction. + Specify the host/guest ABI or reference its versioned specification; + JSON authoring syntax must not silently redefine the binary transport. +- Versioning: explicit protocol/schema identifiers, supported-version and + capability handling, and rejection of unsupported features. Define future + evolution without adding compatibility obligations for the prototype. + +Conformance fixtures include accepted and rejected definitions/requests, +expected error codes/paths, normalized definitions, and deterministic +context/encoding vectors. Exercise the same fixtures through Rust and the +Go guest path, plus SQL extraction where relevant. Randomized ciphertext +comparisons use cross-decryption and query behavior, not byte equality. +Schema validation is one layer of the suite; semantic rejection tests are +required even for structurally valid JSON. + +Keep a single maintained contract: prefer generating structural schemas and +capability metadata from the shared declaration models/catalog where that +faithfully represents the specified grammar, with committed artifacts and +CI drift checks. Review generated schemas against normative rules rather +than treating Serde derives as a complete specification. Do not introduce +OpenAPI or a new transport solely to document this non-HTTP interface. + +## Coverage of the complete review + +The configured-target work above addresses issue 13 in depth. The full API +redesign also requires the following work. Each row identifies a concrete +deliverable; recommendations do not silently settle the open decisions. + +| Work area | Review issues | Required deliverable | +| --- | --- | --- | +| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One signature/import/receiver map, operation-trait design, and dependency migration | +| Context contract | 1, 3, 4, 11, 12, 14 | Scalar and record context rules, explicit optionality, and matching dynamic representation | +| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde/Base64 behavior, and a defined boundary with structural trees | +| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | +| Mixed records and collections | 7, 8, 11 | Passthrough/renaming/context-field derive support and retained collection batching | +| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion and Go coverage | +| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | +| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348–#1352 with this contract; reuse compatible implementation work | +| Prior query design | 16 | Reuse the supplied field-owned query model; reconcile handles, configuration ownership, JSON capabilities, and portability | +| Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | +| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | + +### A. Settle the public method and trait structure first + +Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. +Use the accepted destination-directed decrypt pair. Write +out all signatures, imports, bounds, and return types before renaming traits. +Retain ordinary keyset `decrypt` / `decrypt_with_context` for opaque +ciphertexts, and remove redundant prototype conveniences without +compatibility shims. + +Separate plaintext representation, target operation declarations, and +execution conveniences explicitly. Resolve why `EncryptFrom` declares +`encryption()` while `encrypt_from` lives on `EncryptInto`, and why +`DecryptFrom` supplies methods while `DecryptInto` describes opening. A +rename without an ownership/signature map leaves the discovery problem. + +Recommendation: Stack owns the public value-encryption model; Vitamin C +retains general primitives. Specify the dependency boundary before moving +code, retaining one canonical trait identity and the no-HTTP/WASM builds. +Reassess the original Vitamin C cipher-trait mismatch in that architecture: +do not add a public trait implementation merely for symmetry if it cannot +carry structured KMS context or support its advertised operations. + +Completion: representative scalar, blob, typed record, searchable record, +and JSON calls compile with the documented imports, including generic +callers. The method inventory has no unexplained duplicate execution path. +Compile-fail tests cover genuinely unsupported trait capabilities. + +### B. Turn the context scenarios into a normative contract + +The nine scenarios in the context document are the acceptance matrix, not +optional examples. Cover scalar static context plus caller extension, +source-field selection, caller input, output storage, and decrypt recovery +or expected-context checks. Preserve the accepted `field = "id"` shorthand +as selection on both inputs, without implicit copying. + +Fix composition order and grouping once: literal/field base, composite row +context, and any caller extension. Specify the exact descriptor and AAD +encoding for each case. An expected context checks stored context; it is +not another extension. Never need to open ciphertext to discover the +context required to open that ciphertext. + +Recommendation: represent no input, required input, optional extension, +and optional expectation as explicit capabilities. Do not infer optionality +from the application's `Default` implementation. Settle the wrapper or +trait mechanism alongside the public signatures. Resolve the existing +empty-context mismatch between ordinary scalar encryption and ciphertext +targets without weakening nonempty-context requirements for search targets. + +Give each source/storage/composition rule a dynamic representation and Go +authoring path. Current EQL identifier constraints must be reconciled with +row/caller context explicitly, not bypassed or silently dropped. + +**Next discussion — recommendation, not yet a decision:** distinguish +ciphertext binding from search scope. A row ID may bind a ciphertext to its +record without making every index term row-specific. The target definition +should explicitly describe both scopes; query construction uses the same +declared search scope as index generation. For example, a searchable email +may bind its ciphertext to `(users.email, tenant_id, user_id)` while its +equality index is scoped to `(users.email, tenant_id)`, allowing a query +before the user ID is known. Do not silently remove components from one +generic context argument. Index-key derivation/KMS descriptors must respect +this separation too; changing only leaf AAD is not a complete design. +Finalize the conceptual split before choosing attribute names or exact +tuple encodings, and represent it in the dynamic definition as well. + +**User direction:** allow row-bound ciphertext and document that reads must +provide the row ID to decrypt. Search scope should cover multiple rows, +such as a tenant; automatically including a row ID defeats ordinary search. + +**Misuse-resistant API recommendation (not yet accepted):** + +- Give existing record `context(field = "id")` and its directional variants + ciphertext-binding semantics. Do not implicitly feed those selected fields + or caller extensions into index keys or terms. This is a proposed + clarification for searchable targets; older examples did not distinguish + the roles, and must be updated together before implementation. +- Declare search scope separately on the shared target definition. A named + application scope input (for example `TenantScope`) can come from operation + configuration/caller input. The capability proposal below also permits + record-field selectors when their types explicitly qualify as scopes. + The same typed scope is required for both + writing index terms and constructing queries. A tenant-bound cipher may + supply it only through an explicit declared mapping; never guess it from + unrelated metadata. +- A binding input and a search-scope input are distinct capabilities even + when their underlying values have the same Rust type. Do not provide + blanket implicit conversion between the roles. If one operation needs both + caller inputs, use named members in its typed input rather than one + unlabelled value routed to both operations. +- Missing required scope prevents awaiting a Rust request; dynamic/Go + validation rejects it before key I/O. A default column-wide scope is not + substituted when a tenant scope is missing. Scope-free searches use an + explicitly declared column-wide policy, with column identity still bound. +- The effective ciphertext binding should include the declared search scope + as well as record binding, so changing a supplied tenant also prevents + opening the ciphertext. Index derivation sees column identity plus search + scope only. Query requests never require record binding input. +- Encrypted output must expose whatever the decrypt selector needs or + require it from the caller. Record-projection examples must select the ID; + there is no decrypt fallback which drops row binding. Selecting a field + as context does not implicitly store that field. + +Working declaration spelling: +`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. +The spelling is illustrative; it preserves the agreed directional-context +shorthand and gives search an explicitly separate role. Dynamic definitions +must preserve this separation rather than providing a shared context bag. + +This prevents accidental propagation of row IDs into search terms; it +cannot prove the application chose a scope containing multiple rows. A user +can deliberately put a row ID inside `TenantScope`. Do not claim otherwise, +or attempt to infer semantics from field names such as `id`. Use explicit +types/declarations and documentation for that remaining application choice. + +#### Scope capability and role wrappers — assessment + +**Proposal under discussion:** use both ideas together. Scope is the subset +of context permitted for search terms; semantic wrappers provide positive +trait implementations identifying that subset. Working capability names: +`CiphertextContext` and `SearchScope: CiphertextContext`. These names describe +eligibility, not source selection, optionality, or encoding. Integrate with +the existing context representation rather than introducing a competing +serializer. The previously agreed `ContextInput` capabilities still describe +whether the caller must supply a value. + +For example, `PrimaryKey` implements ciphertext-context eligibility but +not search-scope eligibility. `ScopeKey` implements both, subject to +the inner value's supported context encoding. Generic parameters are Rust +types (`PrimaryKey`); `PrimaryKey(42)` is a value constructor. Prefer +`ScopeKey` as the name for the second wrapper; the user accepted this +name in preference to `Collection`, which can read as a container. +These wrappers need the appropriate `Zeroize` support when they are owned +plaintext fields, consistent with the existing ownership decision. + +Crucially, neither wrapper automatically adds the field to context or scope, +stores it in encrypted output, or changes the plaintext's encryption layout. +The target declaration still selects sources. An illustrative declaration +is `context(field = "id"), search_scope(field = "tenant_id")`. With +`id: PrimaryKey` and `tenant_id: ScopeKey`, the selection is +valid; selecting `id` as search scope is a compile-time error. This makes +typed field selection a reasonable alternative to the earlier recommendation +to accept only caller-supplied search scope. + +Misuse-resistance requirements: + +- No blanket `SearchScope` implementation for every context, serializable + type, raw number, or raw string. Scope use is an explicit classification. +- Composite scopes qualify only when every component is a scope. A tuple + containing a primary key cannot become a scope merely by tuple composition. + Context composites may contain either category. +- Do not forward scope eligibility through `Deref`, `AsRef`, or implicit + conversions that erase a primary-key wrapper. Typed target/query handles + retain the definition's concrete scope type, rather than accepting any + type implementing `SearchScope` interchangeably. Distinct application + newtypes can distinguish tenant and project scopes with equal wire types. +- Permit explicit application opt-in through a documented trait/derive + contract. This is a semantic capability, not an unsafe-memory contract. + Sealing every implementation cannot prove that a value groups multiple + records: a caller can still explicitly rewrap a row identifier as a scope. + Guarantee rejection of accidental role mixing, not business semantics. + +Dynamic definitions must retain field roles and context-versus-scope +selection. Validate those roles before execution: an explicitly declared +primary-key field is not scope-eligible. Plain JSON numbers carry no such +role by themselves; do not infer it from the field name or wire encoding. +Generated Go types can represent the distinction, while the engine validates +dynamic declarations. Rust and dynamic lowering must produce the same +canonical context/scope values; never use Rust type names as wire domains. +Scope classification is not evidence of authorization to access a tenant. + +Acceptance work: valid typed field/caller scopes, rejected primary-key and +mixed-tuple scopes, no automatic selection of marked fields, exact scope +type agreement between write/query, and equivalent dynamic validation. + +#### Dynamic scope roles and validation + +**Recommended design:** declare semantic types and source selectors in the +definition, validate once on load, and supply values separately per operation. +Do not let an individual request assert `scope_key` on an arbitrary value +to override the loaded definition. A definition author can explicitly +classify values, just as a Rust author can construct wrappers; this is +misuse resistance, not authorization or proof of multi-row cardinality. + +**Accepted clarification:** `types` is optional; it holds reusable named +types. Fields may use built-ins or inline role wrappers. Ordinary built-in +or named values need no `kind`. Supported primitives remain eligible for +ciphertext context, but search eligibility requires explicit `scope_key` +classification (or another explicitly supported scope-capable type). + +Illustrative declaration fragment (proposed schema, not today's wire API): + +```json +{ + "types": { + "UserId": { "kind": "primary_key", "value": "u64" }, + "TenantId": { "kind": "scope_key", "value": "u64" } + }, + "fields": { + "id": { "type": "UserId", "passthrough": true }, + "tenant_id": { "type": "TenantId", "passthrough": true } + }, + "context": { "field": "id" }, + "search_scope": { "field": "tenant_id" } +} +``` + +This is the context-related portion of a record target definition; encrypted +email targets and their search options are declared separately as usual. +The shared `field` shorthand selects plaintext fields on encrypt and stored +fields on decrypt; it does not imply a field exists in a query request. +For a query on the derived email handle, the definition exposes a required +`TenantId` scope argument. No `UserId` argument is required or accepted as +an alternative scope. A named/composite scope uses the same ordered schema +on the write and query paths, regardless of their different source selectors. + +Definition loading resolves field references and source/output types, +computes the context/scope capabilities, verifies all composite components, +and rejects a primary-key search selector before key I/O. It also verifies +decrypt selectors are available from stored fields or declared caller input. +Compile the validated definition into an immutable handle shared by encrypt, +decrypt and query operations. No second query-specific scope policy. + +At execution validate presence, nominal type when supplied by a typed handle, +physical representation, range, and composite shape against the selected +definition. Raw wire values obtain their expected semantic type from the +definition; they cannot prove their application provenance. Unknown role +overrides or mismatched nominal value handles are rejected. No fallback to +unscoped querying when a required scope is absent. When JSON is the authoring +or request format, preserve full-width integers through lossless parsing; +the frame transport already distinguishes supported integer widths. + +Recommended Go generation uses distinct named ID types and query methods +with the required scope argument (`TenantID` rather than `uint64` or an +untyped context map). Generated wrappers remain conveniences over the same +validated runtime definition; ungenerated FFI callers receive runtime errors +for invalid declarations/requests. Do not assert Go shares Rust's ownership +or trait-bound guarantees. + +Lower the chosen scope and binding through the existing canonical context +machinery, retaining scalar widths, ordering and grouping. Current dynamic +`context.rs` proves parity between runtime `ContextPiece` trees and Rust +context derivations, but has no primary-key/scope-role distinction. Add role +validation before lowering erases those distinctions, and keep the separate +resolved scope and binding available to their respective operations. +Do not persist role labels in every row just to recover them at runtime; +definition identity and deployment remain explicit concerns of loading the +same policy. Add Rust/Go/dynamic fixtures for valid scopes, primary-key +refusals, nested scopes, missing inputs and identical derived terms/AAD. + +Completion: all nine cases execute through Rust and dynamic declarations; +required input is enforced, altered/wrong context is refused, and existing +context encodings agree across all new implementations. + +### C. Finish the ciphertext and whole-value storage contracts + +Accepted direction: a single-value plaintext encoding produces one opaque +`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode +and seal it as one value, or an explicit target for field-level encryption; +Serde alone does not imply automatic blob serialization. A custom type can +explicitly implement the single-value encoding capability. The wrapper +name and detailed persistence contract remain proposals. +Recommend canonical byte parsing/encoding, human-readable Serde as Base64 +of those bytes, binary Serde as +bytes, and safe Debug behavior. Parsing an envelope does not authenticate +it; authentication occurs during decrypt. + +The existing encrypted-output transcoder can implement the adapter: +`Node::Single(SealedValue)` is consumed by `Visitor::sealed` and moved into +the opaque wrapper. Keep the sealed value internally rather than forcing +a byte serialization at construction. There is no new encryption or KMS +request. On decrypt the wrapper delegates to the same leaf-opening path, +using a single-node wrapper if the internal API requires it. + +Do not convert maps, sequences, passthrough nodes, or sealed empty-container +markers into this type merely because they have a byte representation. +Serializing a whole encrypted tree retains its multiple ciphertexts and +authentication boundaries; it does not produce single-value encryption. +The leaf visitor's unsupported-shape errors are a defensive check, not the +desired public discovery mechanism. Constrain ordinary encryption to +single-value encodable inputs so unsupported structural inputs are rejected +before encryption, statically in Rust where possible and during dynamic +validation. Settle the exact plaintext capability with the representation +traits; do not discover an unsupported shape only after minting keys. + +Keep generic structural encryption and searchable JSON explicit. Specify +whether structural-tree persistence is supported and under what stable +format. Any protocol changes must be coordinated with their consumers. + +#### Typed encodings and extractable ZeroKMS tags + +**Requirements added, 2026-10-25:** ciphertext text conversion returns the +existing concrete encoding wrappers, not `String`. The user also wants to +explore selecting an encoded ciphertext as a field's output type. The +ZeroKMS key tag must be easily extractable from stored ciphertext, including +in PL/pgSQL, to support a separately designed refinable-tag mechanism. +The storage/authentication contract is not yet locked in. + +Current `eql-bindings::encoding::{Base64Text, Base85Text}` validates canonical +padded standard Base64 and the crate's RFC 1924 Base85 encoding respectively. +Their private strings, encoding constructors and validating deserializers +make decoding infallible. They guarantee the text encoding, not that the +bytes form a ciphertext. Reuse these types and add ordinary fallible text +parsing conveniences rather than requiring Serde to construct them from text. + +Recommended ownership: a small Vitamin C encoding component, re-exported by +Stack Encrypt and EQL. These byte/text utilities do not depend on encryption, +SQL or EQL layouts. Keep TypeScript/schema integration optional in the owning +crate, accounting for Rust's orphan rules when moving the existing trait +implementations. Do not introduce a Stack Encrypt dependency on EQL merely +to reuse its text wrappers. Final component placement remains a proposal. + +Proposed ciphertext methods return `Base64Text` / `Base85Text` from +`to_base64` / `to_base85`; corresponding constructors accept the typed text +and still return a ciphertext parse error when its decoded envelope is +invalid. Encoding validation, envelope parsing and authentication are three +different guarantees. Use one canonical byte envelope under both encodings. + +**Accepted, 2026-10-25:** support explicit encoded ciphertext field targets. +Use a ciphertext-specific storage wrapper +(working names `Base64Ciphertext` / `Base85Ciphertext`) over making every +`Base64Text` an encryption target. The wrapper validates both text and +envelope, serializes as the text string, and composes with `EncryptTarget` +and `DecryptSource`: seal once, then encode; parse before opening. These +names remain provisional until the final naming pass; the first-class +encoded target capability is settled. A structural output +can mix `Ciphertext` and encoded ciphertext fields without additional KMS +calls. Do not silently change the prescribed representation of EQL targets. + +The current `SealedValue` v1 envelope already uses a simple prefix: +`version(1) | keyset_id(16) | iv(16) | tag_len(2, LE) | tag | local_ciphertext`. +Its ZeroKMS tag begins at byte offset 35, with length at offsets 33–34. +This is evidence that extraction can be simple, not a commitment to keep +those offsets. Keep the chosen version's layout documented and parseable +without the plaintext codec, key retrieval, or decryption. Prefer a checked +borrowed header/envelope view and an accessor explicitly named `key_tag` +over decomposing the whole ciphertext or conflating it with the AEAD tag. +Extraction does not authenticate the returned tag. + +Ship a matching SQL/PL/pgSQL extraction helper and Rust/Go/SQL fixtures with +the storage format. Check versions, lengths and truncation before slicing. +For text storage prefer Base64 when SQL must inspect the header: +[PostgreSQL](https://www.postgresql.org/docs/current/functions-binarystring.html) +has built-in Base64 decoding and byte slicing, but not Base85 decoding. +If SQL extraction from a Base85 envelope is supported, supply an explicit +decoder; do not leave it as an assumed capability. PostgreSQL Base64 output +also inserts line breaks, unlike the canonical Rust wrapper: normalize SQL +output or validate it against the same canonical rules in cross-language +tests. JSON-tree storage needs its own header adapter: its shared key header +is distinct from each node's ciphertext and parent tag. + +The proposed refinement uses the ZeroKMS tag as an HMAC key and replaces it +with a derived tag. **User clarification:** at decryption, ZeroKMS uses the +refined tag for policy checks and logging, and returns the original tag. +The decrypt path uses that returned original tag to reconstruct leaf AAD. +This resolves the conceptual conflict with binding the original tag into +AEAD: the stored/request tag and the authentication tag input need not be +identical after refinement. Keep those roles explicit in the KMS response +and decrypt implementation. Tag extraction returns the tag actually stored +in the envelope, which may be refined. Protocol details remain deferred; +this is the intended behavior, not a claim that the current client/engine +already implements it. Add refinement/decrypt integration coverage when +that work lands. + +**Decision, 2026-10-25:** direct `encrypt` handles one single-value input; +`EncryptInto` handles collection composition and batching. This supersedes +the earlier open question about automatic collection lifting. The agreed +public examples are: + +```rust +let scalar: Ciphertext = keyset.encrypt(1_i32).await?; + +let batch: Vec = vec![1_i32, 3_i32] + .encrypt_into(&keyset) + .await?; + +let blob: Ciphertext = keyset + .encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])) + .await?; +``` + +The batch produces two independent ciphertexts through the shared batch +executor; the blob encodes and seals the complete collection as one value. +Do not add implicit collection handling to direct `encrypt`, or a separate +batch entry point merely to provide functionality already covered by +`EncryptInto`. Context-taking variants follow the same division. + +Acceptance checks: reject direct `encrypt(Vec)` before execution; +support context-free `Vec` to `Vec`; preserve target-directed +collection opening and batching; and recover the blob as a whole collection. +Existing collection composition is retained, while opaque `Ciphertext` and +its no-context target support are new work. + +The current prototype treats `Vec` as one byte value, but that is not a +compatibility constraint. The latest recommendation is uniform collection +behavior: `Vec` to `Vec` encrypts individual bytes, while +`OpaqueBlob>` encodes the complete vector as one value. Do not add +`Vec` to `Ciphertext` conversion or a bytes wrapper solely to retain +prototype behavior. A distinct raw-byte capability would need its own use +case. The blob codec need not reproduce the prototype's raw-byte encoding. + +For `OpaqueBlob`, select the library-owned codec, authenticated codec/type +discriminator and version, schema-evolution policy, error behavior, and +supported source/decoded types. Authenticate before deserializing plaintext. +Do not silently reinterpret an ordinary byte ciphertext as a different +encoding. Preserve an explicit wrapper rather than implicit Serde fallback. + +The constant-time question needs a concrete conclusion: assess `serdect`, +the codec, both sides of Serde (value traits and serializer/deserializer), +length/shape disclosure, and permitted custom implementations. If arbitrary +Serde types remain supported, do not promise end-to-end constant-time +serialization. Select a restricted profile separately if that property is +required; it cannot be inferred from the wrapper or its encryption primitive. + +Completion: scalar and blob examples persist and reload through a fresh +cipher, reject invalid encodings, and verify fixtures for the selected +format. The supported timing contract and codec version are documented. + +### D. Complete record derives and preserve collection composition + +Implement explicit passthrough and source-field renaming, independently of +context selection and storage. Generate decrypt reconstruction for the +same record. Support nested targets with and without configuration, and +report incompatible context or source types at their declarations. + +Keep the collection behavior already present: `Vec` to `Vec` composes +the target operations into one logical batch, rather than awaiting every +row separately. Preserve document/leaf key granularity; one logical batch +does not promise one physical ZeroKMS request beyond backend limits. + +Completion: the public mixed-record and searchable-record examples run, +including batch encrypt/decrypt, nested targets, renamed passthrough fields, +and stored context. Dynamic/Go declarations produce equivalent semantics. +A plaintext-side derive generating output types is deferred convenience, +not a prerequisite for these workflows. + +### E. Finish the configured policy's lifetime and query surface + +Use the prior query-design baseline and reconciliation above. The query +model is not an empty design space: field-owned index selection, typed +handles, and shared derivation already have a detailed proposal. + +The typed-factory direction is established. The target-definition section +above now proposes its owner: a shared immutable fixed definition, or an +explicit runtime instance, with generated typed field handles. Review that +proposal and prove its initialization/lifetime behavior before treating the +once-only reuse claim as implemented. Multi-field runtime configuration and +the exact generated signatures still require specification. + +Write complete static and Go examples for equality, JSON containment, +ordered paths, and extraction, all using the write policy. Specify query +results and unsupported-operation errors. Keep those examples free of +mandatory hand-written plans. Preserve direct typed EQL conversion and +catalog/guest support as deliverables of the configured-target sequence. + +### F. Make naming and coordinated integration part of completion + +Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for the +proposed role-based JSON names, retained `OpaqueBlob`/`Ciphertext` spellings, +shared format crate and primitive/application split. Retain accepted +declaration/convenience trait names. Recommend renaming the plan builder's +output-selection method to `target`; distinguish it from executing +`EncryptInto`. New spellings and crate moves still require review. + +Remove replaced APIs directly and update EQL, Go, and guest consumers +together. There is no requirement to read prototype ciphertext or preserve +old imports. Specify the selected encoding and cryptographic semantics, +and establish fixtures for that contract going forward. + +Completion: all public examples are executable tests, static/dynamic +cross-opening and querying pass against the selected storage format, batching +checks pass, and the guest builds remain supported. No new API is described +as implemented until its Rust and Go acceptance criteria are met. + +## Recommended order for the remaining design decisions + +### Decision checkpoint, 2026-10-25 + +Settled direction: keyset-directed single-value encryption; target-directed +collection batching; explicit `OpaqueBlob`; conversion methods separate +from `EncryptTarget`/`DecryptSource` declarations; explicit context-input +capabilities instead of `Default`; typed Rust options factories and dynamic +deserialization into the same options; static and dynamic support together; +shared typed definitions and field handles; a consistent `using` selector +with strict typestate and definition-first query spelling; `OpaqueEncode` +/ `OpaqueDecode`; consuming encrypt and encrypt_into with guarded cleanup; +ordinary keyset decrypt alongside the destination-directed decrypt pair, +both with context variants and ordinary owned `T: Zeroize` results; +separate ciphertext context and search scope with `PrimaryKey`/`ScopeKey` +eligibility; definition-time role validation in the dynamic interface, +optional named-type declarations and explicit scope eligibility; and a +versioned interoperability specification with shared conformance coverage; +and no backward compatibility with the prototype. Do not reopen these +choices merely because older examples or work-area descriptions retain +provisional wording. + +The remaining product/API decisions are narrower. Distinguish them from +specification and implementation obligations: + +| Remaining question | Decision required | +| --- | --- | +| JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | +| Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | +| Stable blob profile | Specify supported Serde shapes, lossless options/enums/maps, authenticated codec versions and compatibility fixtures under Vitamin C #519. This is the immediate priority; format-preserving cleanup/timing improvements are follow-up work, with limitations documented honestly. | +| Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | + +Next review the JSON operation document and its static/dynamic examples, +then turn the required inventory into implementation phases and formal +interoperability fixtures. Stable storage-format work is tracked upstream. +Finish remaining query integration details without reopening ORE equality. +The user has now requested the naming/package pass while JSON review +continues; that proposal is ready for review in the packaging companion. + +Specification/implementation work still includes exact context/scope grouping +and encoding, dynamic grammar and version identifiers, error codes, binary +envelope fields, Rust/Go/SQL fixtures, and the refined-versus-original tag +flow. Record candidate choices in the formal specification for review; do +not treat every byte offset or adapter signature as a new architectural +decision. The initial persistence recommendation remains ordinary Serde +composition for explicit layouts, without a new arbitrary-tree wire format. + +Cleanup implementation still needs guarded encoder sinks, partial decode +handling, cancellation/drop checks, and an explicit drop-guarded result +adapter. Under the user's latest sequencing decision, implementation +improvements may follow the stable format; preserve the stored contract and +document actual guarantees instead of claiming the eventual goals are +already satisfied. No completed constant-time/zeroization audit is implied. + +The shared-definition caching, initialization error/retry behavior, generic +identity, and request lifetimes still need concrete signatures and proof. +That is implementation design within the recorded ownership model, not a +reason to reopen the question of whether writes and queries share a policy. + +Passthrough implementation, transcoder adapters, collection composition, +trait relocation, documentation updates, and acceptance tests remain work +items. They are not reasons to characterize the agreed API direction as +undecided. No backward-compatibility phase is needed for the prototype. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index ad6afc78e..f60a088e5 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -7,12 +7,20 @@ Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-e an API design draft covering encryption, persistence, decryption, and queries. The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) explore context sources and storage using Serde-inspired attribute syntax. +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +requirements, recommended decisions, sequencing, and completion criteria. This is a running issue log. Add a numbered section as each issue is found; keep observations, proposed changes, and unresolved decisions distinct. All issues below remain unimplemented. Individual decisions are recorded within each issue; proposed APIs are illustrative, not implemented. +**User clarification, 2026-10-25:** this API has never shipped beyond an +early prototype. Earlier suggestions to preserve old encodings or public +imports are superseded: backward compatibility and prototype data migration +are not requirements. New Rust, Go, guest, and EQL implementations must +agree with each other, but need not retain the prototype's representations. + ## 1. Cipher trait mismatch and explicit context ergonomics **Status:** Open. @@ -130,6 +138,22 @@ the simple scalar case. ### Proposed direction and open decisions +**Subsequent decision:** ordinary encryption should produce one opaque +ciphertext; choose structural encryption through an explicit target, or +encode a composite plaintext first with `OpaqueBlob`. The +[implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends +adapting a native single sealed leaf through the existing transcoder, +without serializing an entire encrypted tree or re-encrypting it. + +**Collection decision, 2026-10-25:** direct `encrypt` handles a single-value +input and does not automatically lift over collections. Retain collection +composition through `EncryptInto`, including +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +Use `OpaqueBlob>` to encode and seal a complete collection as one +ciphertext. This settles the previously reopened collection question. +The spelling for byte buffers (`Vec` versus an explicit bytes wrapper) +remains a separate detail to resolve. + Provide a straightforward opaque ciphertext storage round trip for scalar values, with direct byte conversion and reconstruction. Decide whether this means a dedicated scalar result type, a wrapper, or conversion helpers on @@ -138,8 +162,19 @@ the existing API. Also decide whether Base64 and Serde should be supported conveniences, and whether whole-tree serialization belongs in Stack or remains caller-owned. If it remains caller-owned, make that distinction explicit and keep it out -of the simple scalar path. Preserve the existing canonical leaf encoding; -an ergonomic improvement should not require changing stored ciphertext. +of the simple scalar path. The prototype's canonical leaf encoding is a +useful starting point, not a backward-compatibility obligation. + +**Follow-up, 2026-10-25:** text conversions must use the existing +`Base64Text` / `Base85Text` wrappers from EQL bindings. Explore moving these +general encoding types below Stack/EQL and supporting ciphertext-specific +encoded field targets. The persisted ZeroKMS key tag must be extractable +simply in Rust and PL/pgSQL. A refinable-tag protocol is being designed +separately. The user clarified that ZeroKMS checks policy/logs against the +refined tag and returns the original tag for reconstruction of leaf AAD; +record these distinct roles in the implementation. The +[persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) +record the proposals and the Base85/SQL decoding constraint. **Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) (`StackCipherText`, `SealedValue`) and @@ -540,11 +575,10 @@ become another public API users must understand. Move the value-level backend adapters too, or separate a lower-level core/primitive boundary. - Context composition, type tags, ciphertext framing, index normalization, - and binding transport frames have compatibility consequences. Preserve - their existing encodings when changing ownership; any deliberate semantic - change needs separate versioning and migration treatment. -- Retain one canonical trait/type identity. Compatibility re-exports can - preserve an old path where dependencies permit; copied independent trait + and binding transport frames must agree across the new consumers. They + may change without preserving prototype data or providing migration. +- Retain one canonical trait/type identity. Update consumers directly; + compatibility re-exports are unnecessary. Copied independent trait definitions would require duplicate implementations and adapters. - Account for guest/FFI consumers and EQL target implementations, not just Rust application imports. The no-HTTP/WASM path must remain usable. @@ -682,3 +716,408 @@ for the proposed selectors and input requirements. **Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) (`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). + +## 13. JSON layouts and encrypted-output adaptation + +**Status:** Open; initial assessment of the JSON tree implementation on +`next` at `b1049824b`. No replacement API selected. + +**Agreed follow-up:** investigate replacing the EQL JSON adapter's +serialization round trips with direct typed conversions. This does not yet +select an extended visitor protocol or authorize a particular replacement +API. + +`Layout` describes a target selected by a value carrying runtime options. +It associates a `Target: EncryptFrom` and an `Index`, and supplies an +`encryption(&self)` description plus the configured index for queries. +`EncryptFrom::encryption()` instead selects a description by output type. +For JSON, both routes can produce `JsonDocument`: selecting that type uses +default options, while `Json::new(options)` can specify ordered paths and +whether the root is sealed. A plan retains those options for its queries. + +This is broader than choosing a Rust record's field layout. JSON options +affect cryptographic structure, query capability, and information exposed +to the server. The same output type can represent different configurations; +the type alone does not identify them. The JSON tree belongs to the +searchable-layout workflow, rather than being an opaque serialized blob. + +The existing `target::transcode` protocol serves a different stage: a +`Reader` hands completed ciphertext, structural markers, metadata, and terms +to a destination `Visitor`. `Encryption::transcode()` uses it to adapt +encrypted output without an intermediate serialization tree. It does not +choose JSON indexing options or generate missing search terms. + +The new EQL adapter currently takes another route. `JsonTree` implements +`Layout`, delegates encryption to the +engine's `Json` layout, and wraps its completed `JsonDocument` with EQL +version and column metadata. Its envelope conversion uses a +`serde_json::Value` intermediate; a private helper also named `transcode` +converts query/extraction types through `to_value`/`from_value`. Neither is +the existing `target::transcode` protocol. That protocol currently has no +JSON-document reader or visitor event. The conversion processes encrypted +output and query metadata; it is not a plaintext serialization bypass. + +On decrypt, the EQL target validates its stored column against any supplied +expectation, reconstructs the engine document through the wire form, and +delegates opening to the engine. Its layout object is not required to +perform that conversion. + +### Questions to resolve + +- Keep encryption configuration distinct from encrypted-output adaptation. + Decide whether `Layout` is the clearest public name for a configured + target, especially alongside the earlier use of "layout" for output + records. +- Decide how runtime target options fit the retained `EncryptInto` API; + currently type-directed conversion selects the target's defaults, while + plans accept a configured layout via `encrypt_with_layout`. +- Evaluate a direct typed JSON-document adapter versus extending the + encrypted-output visitor. Avoid forcing the JSON protocol into generic + map/sequence events that lose its header, shared document key, projection, + and extraction semantics. A shared architectural boundary does not + necessarily require one universal visitor. +- Consider whether a general configured-target abstraction should require + an index: today's `Layout` requires one even though encryption-only + targets can also need options. +- Preserve the link between write configuration and query configuration; + a plan's reuse of the configured index is useful independently of naming. + +**Sources:** [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) +(`Layout`, `json_document`, `EncryptFrom for JsonDocument`), +[encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs), +[JSON options and document](../../packages/stack-encrypt/src/json/mod.rs), +and [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) +(`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). + +### Static Rust surface versus runtime configuration + +The intended public Rust experience is traits and derives. Treat plans as +the lower-level composition interface supporting those APIs and the dynamic +FFI interface, rather than requiring Rust callers to write a plan for JSON. +The derive already emits plans; its implementation can retain that shared +execution path without making the plan builder the primary user example. + +JSON already has a type-directed path. Given an initialized `cipher` and +its `keyset`, this uses existing methods and default JSON options: + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use eql_bindings::Identifier; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, EncryptInto}; + +let profile = json!({"name": "Alice", "age": 34}); +let column = Identifier::for_column("users", "profile")?; + +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored: Value = encrypted.decrypt_into(&keyset, column).await?; +assert_eq!(restored, profile); +``` + +This produces searchable JSON with a sealed root and no ordered paths. +The explicit decrypt context checks the expected column against the stored +identifier. The snippet is source-checked, not compiled here. + +The earlier `SearchableEmail` example fixes its recipe in the target type: +ciphertext plus an equality term. `EncryptFrom::encryption()` takes no +configuration value. JSON's default target also fixes a recipe in its type. +What `Layout::encryption(&self)` adds is access to instance configuration: +one `JsonTreeDocument` output type can be produced with `$.age` ordered, +with `$.salary` ordered, or with different root settings, selected at +runtime. Its configured index also supplies the matching query behavior. + +Search metadata itself does not require `Layout`. A custom target could +fix ordered JSON paths in a manual `EncryptFrom` implementation using the +existing JSON operation. A derive could eventually generate such a fixed +policy. Runtime-configured targets need an additional configuration input; +today `encrypt_into_with_context` accepts context but no layout/options, +and custom JSON configurations use the lower-level plan or `keyset.run` +path. Keep those options separate from cryptographic context. + +The cryptographic distinction is separate from the API distinction: JSON +uses a specialized document operation with a shared document data key and +node/extraction semantics. It cannot be implemented merely by adding a +generic equality-term field to the ordinary ciphertext tree. That explains +the JSON operation, but does not by itself require a public `Layout` trait. + +### Naming obscures roles + +| Current type | Role | +| --- | --- | +| `serde_json::Value` | Plaintext input and recovered plaintext | +| `stack_encrypt::json::JsonOptions` | Runtime configuration | +| `stack_encrypt::target::Json` | Configured index; also implements `Layout` | +| `stack_encrypt::json::JsonDocument` | Encrypted engine document, including search metadata | +| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target/layout | +| `eql_bindings::v4::json_tree::JsonTreeDocument` | Encrypted EQL storage envelope | + +Review names that state these roles, using `Encrypted` for ciphertext-bearing +documents, `Index` for index descriptions, and `Target` or `Config` for +configuration, with module paths distinguishing EQL from engine types. +No replacement names are selected. `Json`'s dual index/layout role should +be considered before simply renaming it. + +### Candidate: a fixed-policy wrapper around the EQL output + +**Proposal, not implemented or agreed syntax.** Attach the fixed policy to +an output newtype, while the inner library type owns the database format. +The alias below only clarifies the existing type's role; it is not a +library rename. + +The later [implementation plan](2026-10-24-stack-encrypt-api-plan.md) +recommends a general typed-options factory, `target(options = factory)`, +instead of the JSON-specific attribute below. Dynamic callers deserialize +options into the same concrete type; Rust and Go support ship together. +This sketch is retained to explain the earlier alternative. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use eql_bindings::Identifier; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, DecryptInto, EncryptFrom, EncryptInto}; + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = Value, + transparent, + json_tree(ordering_paths = ["$.salary"]) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); + +// Given a keyset: +let payroll = json!({"name": "Alice", "salary": 120_000}); +let column = Identifier::for_column("employees", "payroll")?; + +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +// The wire representation is the inner EQL envelope, with no extra field. +let wire = serde_json::to_vec(&encrypted)?; +let loaded: EncryptedPayroll = serde_json::from_slice(&wire)?; +let restored: Value = loaded.decrypt_into(&keyset, column).await?; +assert_eq!(restored, payroll); +``` + +The proposed `stash(transparent, json_tree(...))` selects the configured +JSON operation for the inner target and wraps its completed output. It must +not invoke the ordinary scalar/record derivation on the wrapper, nor invoke +the inner target's default encryption and then try to add ordering terms. +It inherits the EQL target's caller-context and decrypt behavior. Serde's +existing `transparent` attribute preserves the EQL envelope; it does not +provide database-driver trait implementations, which remain a separate +integration concern. The root stays sealed under this candidate's defaults. + +The generated declaration must also expose the configured index metadata +for future typed queries, so queries can reuse the policy instead of +repeating paths. The query-facing syntax remains undesigned. The core derive +must delegate configured output construction to a capability of the inner +target, rather than depending directly on the EQL crate. Whether to express +that capability through `Layout` or another trait remains open. + +### Alternative: configuration on a plaintext wrapper + +The user's proposed `JsonPlaintext::new(value).with_ordering_path(path)` can +still yield the exact EQL wire format. Configuration source and output +representation are independent: an explicit `EqlEncryptedJsonTree` target +can accept that input and execute the same configured JSON operation. The +wrapper must be treated as an encryption request, not serialized as part of +the plaintext. The current API does not provide this wrapper. + +This is attractive when paths come from runtime configuration, but repeats +fixed policy at call sites and needs a way to reuse that policy for queries. +Prefer the output newtype for the fixed-policy example. If the runtime +wrapper is added, initially do not implement conversion from it into +fixed-policy targets: avoid silently overriding or merging conflicting +policies. Neither syntax permits callers to redefine EQL's required fields +or its cryptographic wire protocol. + +A fixed Rust type describes the policy for new writes; transparent Serde +deserialization alone does not prove a stored document was produced with +that policy. Ordered paths are not all recorded in the document header. +Policy changes and heterogeneous existing rows need explicit compatibility +and migration semantics before claiming stronger read-time guarantees. + +## 14. Decrypt convenience methods need explicit context variants + +**Required direction:** provide a no-context method and a separate +`_with_context` method, mirroring the encrypt-side distinction. The user +prefers the name `decrypt_from`. + +Today `ciphertext.decrypt_into(cipher, context)` exists on the blanket +`DecryptFrom` extension trait. `Plaintext::decrypt_from(ciphertext, cipher)` +and `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)` also +exist on that trait. `DecryptInto

` separately declares the opening +operation. These method/trait names are easy to confuse. + +**Recommendation:** use the destination-directed pair in the preferred +public examples. `User::decrypt_from(encrypted, &keyset)` follows Rust's +From convention; `encrypted.decrypt_from(...)` would reverse it. The +receiver and migration of the competing method remain recommendations to +confirm, not settled user decisions. Do not change the declaration trait +merely because its convenience spelling changes. Address trait discoverability +with issue 5 and optional-context capability with issue 12. + +## 15. Coordinate the Go JSON work with the shared API redesign + +The user identified [PR #1353](https://github.com/cipherstash/stack/pull/1353) +as parallel agent work. The PR is currently a Go JSON design document with +implementation tasks #1348 through #1352, tracked by #1347. Its dynamic +JSON/options path, public names, context behavior, and staged extraction +support must be reconciled with the configured-target and trait/context +contracts developed here. Preserve compatible catalog, transport, engine, +and testing work rather than designing or implementing it twice. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +the concrete reconciliation points after reading the PR write-up. This is +a local coordination item; no message or review was posted to that agent +or PR. + +## 16. Integrate the prior query design rather than reopening it + +The user supplied **Stack Encrypt query design** (2026-10-21), read in full +on 2026-10-25 from `/Users/dan/Downloads/Stack Encrypt query design.md`. +It proposes field-owned index selection, one answering index per operator, +typed field handles, a single capability declaration, and shared query +derivation/refusals. It is explicitly a discussion draft with open questions. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses that +draft as the baseline and records concrete reconciliation points: our +output-side derives and configured options factories, JSON operators and +extraction, context inputs, static/dynamic portability, and validation +timing. Older compatibility requirements are superseded by the user's +prototype clarification. The source document was left unchanged. + +## 17. Use consistent definition-selection names for encrypt and query + +The user finds `.using(...)` for encryption and `.on(...)` for queries +confusing and asks to consider one name for both. Recommend `.using(...)`: +the selected definition or field supplies the operation's policy. Final +spelling is not yet accepted. Retain the operation-specific bounds: +encryption selects the output target's associated definition; queries +require a field or single-value target implementing `QueryTarget`. +This is a naming change, not a merger of those capabilities. + +The user also prefers `cipher.using(&field).query(op, input)` and explicitly +requires strict typestate. Supporting both orders is acceptable to them. +The plan recommends definition-first as canonical, with operator-first +converging to the same typed request. Missing definition/required context +must prevent awaiting on the static path; repeated selection must not +silently overwrite a request's definition. The current `.query(...).on(...)` +implementation is not claimed to already provide this API. + +## 18. Single-ciphertext representation needs consuming, cleanup-aware inputs + +**Decision, 2026-10-25:** use `OpaqueEncode` / `OpaqueDecode` for single-value +encoding/decoding, distinguishing one opaque ciphertext from target layouts. +Both direct `encrypt` and target-directed `encrypt_into` consume input and +are responsible for cleanup of owned plaintext artifacts, including the +context-taking variants. Explicit passthrough fields move to the output. +The plan records source cleanup capabilities, +immediate guarding before returning an unpolled request, cleanup of encoded +buffers/scratch on all ordinary exit paths, and boundaries for caller copies, +custom serialization, passthrough output, and Go host/guest ownership. +`Serialize` alone cannot establish cleanup of an `OpaqueBlob` source. +Consumption and zeroization are distinct; no completed audit is claimed. + +**Follow-up decision:** plaintext types must implement the existing +`Zeroize` trait; do not introduce our own cleanup trait for now. For the +proposed Serde blob, this means `Serialize + Zeroize` on input and +`DeserializeOwned + Zeroize` on owned decode. Library guards must actually +invoke cleanup; partially constructed Serde values need separate treatment. + +**Decrypt decision:** retain `keyset.decrypt` / `decrypt_with_context` for +ordinary opaque ciphertext and `T::decrypt_from` / +`decrypt_from_with_context` for declared targets. Return an ordinary owned +`T: Zeroize` on success, transferring cleanup responsibility to the caller. +Stack guards plaintext intermediates while it owns them. Explicit +drop-guarded results should be supported without becoming the default; +`Zeroize` alone does not imply cleanup on drop. This does not resolve the +separate partial-deserialization cleanup requirement. + +**Codec recommendation:** reuse Vitamin C frame machinery with a blob +mapping. The 0.8.0 transport Serde implementation already exists and guards +serialization buffers, and its frame writer wipes allocations on growth. +Do not reuse that mapping unchanged: option presence can be lost +(`Some(())` and `None` both encode as null), maps require text keys, and +successful `to_vec` returns an unguarded vector. See the +[codec assessment](2026-10-24-stack-encrypt-api-plan.md#blob-codec-assessment-vitamin-c-frames) +for the required work and Borsh fallback. Neither codec choice nor the +`Zeroize` bound establishes constant-time behavior or partial-decode cleanup. + +**Priority update:** first specify stable, versioned storage bytes; the +existing frame codec was ephemeral transport. Further zeroization and timing +implementation improvements may follow without changing stored format. +The upstream action is +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). + +## 19. First-release completeness and remaining review decisions + +**Accepted, 2026-10-25:** encoded ciphertext wrappers are first-class field +targets; final wrapper names/package placement come last. ORE does not +answer equality initially, even for integers. Preserve a future explicit +capability for equality-preserving encodings; never infer exact equality +from ordering support for text or other lossy encodings. Keep operator +provider overlap validation and avoid silently changing existing definitions. + +The JSON first version must be comprehensive across Rust and dynamic/Go, +including extraction, opening entries, children/counts and subtree walks. +Claude's earlier Go plan deferred walks; the new scope supersedes that +deferral. The [operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) +lists required operations, current evidence, proposed Rust/dynamic surfaces, +SQL use, missing-path/null semantics and implementation acceptance gates. +Names and packaging are explicitly the last design activity. + +**Further accepted decisions:** custom Rust stages can remain explicitly +Rust-only with `NotPortable` for unsupported dynamic export; built-ins retain +Rust/Go parity. All key-independent checks precede key resolution/loading. +The detailed JSON operation contract is still under user review. + +## 20. Plaintext-derived keysets and optional provider policies + +The user requested routing a plaintext record to a keyset by ID/name from +one of its fields; assessment of Vitamin C's unused `AsyncCipher`; and +ZeroKMS lock context including principal JWT claim selectors. Keyset and +lock-context capabilities must remain optional for future AWS KMS/Vault +backends. See the [provider-capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) +for current source evidence, proposed static/dynamic declarations and +acceptance work. + +Current facts: Stack resolves keysets by ID/name but generation batches use +one preselected keyset. Stack deliberately orchestrates provider requests +outside Vitamin C's leaf walk and already implements synchronous `Cipher`. +Its provider payload layer supports lock context but Stack Encrypt sends +empty lock context. The existing data/index key abstractions still embed +ZeroKMS protocol types; future providers need a neutral base plus optional +capabilities, not no-op implementations of mandatory ZeroKMS features. + +Recommendations, not yet accepted: derive key routing as an explicit target +selector and resolve it in the client executor; partition batches by resolved +key/provider/principal requirements; carry lock context as a separate typed +provider policy on both generation and retrieval; reject unsupported required +features before I/O. Keep key routing, ciphertext context, search scope and +principal policy separate. Revisit cipher adapter cleanup during packaging +without discarding cross-record batching or native/WASM support. + +## 21. Packaging, naming and retirement of AsyncCipher + +The user favors removing Vitamin C `AsyncCipher` and moving application +encryption behavior into Stack Encrypt. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes +Stack ownership of value traversal, declarations, target/index adapters and +execution; a small shared `stack-encrypt-format` crate for codecs and typed +encodings; a ZeroKMS-specific client crate; and reusable primitives remaining +in Vitamin C. It includes an acyclic dependency map and public name glossary. + +Stack does not use `AsyncCipher`, but Vitamin C's async helpers and tests do. +Retire that execution path with its consumers and move relevant coverage; +do not treat deletion of the trait alone as complete. Normalization/PRF/ORE +adapters also depend on the existing value model and need separation from +their primitives. Moving the frame implementation refines the earlier codec +ownership recommendation; coordinate stable storage work under Vitamin C +#519. Names and package moves remain proposals, not implementation changes. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md index da361fc04..ee700cabc 100644 --- a/docs/plans/2026-10-24-stack-encrypt-public-examples.md +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -5,6 +5,12 @@ Date: 2026-10-24. Companion to the The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) explore directional source selection, storing context, and decrypt inputs across nine scenarios. Their attributes are proposals, not implemented APIs. +The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) +covers the comprehensive first-release JSON workflows with Rust, dynamic +and SQL examples, including extraction and walks. + +The user clarified on 2026-10-25 that this API is an early prototype: +backward compatibility with its old APIs and encodings is not required. This document tests the proposed public API against three complete workflows: encrypt, persist, load, and decrypt a user; the searchable workflow also @@ -21,6 +27,36 @@ the examples really encode, write, read, and decode their stored forms. ## What is agreed, and what these examples propose +**Ownership decision (2026-10-25):** `encrypt` and `encrypt_into` consume +their inputs, including context-taking variants. Their implementations must +guard owned plaintext and temporary buffers for cleanup on success, error, +and request drop; passthrough fields intentionally transfer to the output. +The single-ciphertext capabilities are named `OpaqueEncode` / `OpaqueDecode`. +Plaintext types must implement `Zeroize`; no new cleanup trait is planned. +The candidate programs below predate that bound: update their source derives +and any reuse of consumed input when making them executable. +`Serialize` alone does not establish the cleanup contract for `OpaqueBlob`. + +**Decrypt decision (2026-10-25):** retain direct `keyset.decrypt` and +`decrypt_with_context` for opaque ciphertext, alongside destination-directed +`T::decrypt_from` and `decrypt_from_with_context` for declared targets. +Successful calls return ordinary owned `T: Zeroize`; the caller then owns +cleanup responsibility. Stack guards intermediate plaintext. Support +explicit drop-guarded results without imposing a wrapper by default; +`Zeroize` alone does not wipe returned values on drop. These are accepted +API contracts, not a claim that the prototype already implements them. + +**Collection decision (2026-10-25):** direct `keyset.encrypt(value)` handles +one single-value input and produces `Ciphertext`; it does not automatically +encrypt collections element by element. Collection batching uses +`EncryptInto`, for example +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` instead produces +one ciphertext for the encoded collection. The latest recommendation +applies the same rule to byte vectors, with no wrapper or special case +needed to preserve prototype behavior. These calls require the proposed +opaque ciphertext and no-context target support. + | Surface | Status in this review | | --- | --- | | `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | @@ -32,10 +68,11 @@ the examples really encode, write, read, and decode their stored forms. | `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | | `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | | `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | -| `Ciphertext` as an encryption/decryption target | Proposed scalar target, including support for `String` and `Vec`; does not silently flatten a struct into a blob | +| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64/Base85 storage compose with encrypt/decrypt targets; final names remain for the naming pass | +| `Ciphertext` as an encryption/decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | | Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | -| `cipher.decrypt_with_context` | Proposed companion to `encrypt_with_context`; ordinary decrypt naming remains a review decision | -| `User::decrypt_from` / `decrypt_from_with_context` | Existing `DecryptFrom` extension conveniences, retained here pending the decrypt naming discussion | +| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair, symmetric with direct encryption | +| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair on `DecryptFrom` | | `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or @@ -46,8 +83,8 @@ subject to review where marked proposed or provisional. The proposed opaque ciphertext is intentionally untyped: the decrypt call's expected type selects decoding. A typed wrapper remains an open alternative. Its byte decoder parses a stored envelope; authentication happens on decrypt. -For a derived field it must preserve the existing context and leaf encoding, -not introduce a new cryptographic format just to improve ergonomics. +Context and leaf encodings must agree across the new Rust and dynamic +implementations; preserving the prototype's encoding is not required. The trait name `DecryptFrom` is somewhat surprising: it supplies the convenience functions, while `#[derive(DecryptInto)]` describes how an @@ -84,8 +121,8 @@ application to select and invoke a codec. The proposed wrapper owns the input; operation. The context identifies the application's document schema and binds this document to an externally known record id. -For this draft the wrapper accepts Serde `Serialize` on write and -`DeserializeOwned` on read. That input-trait choice is provisional: it does +For this draft the wrapper accepts `Serialize + Zeroize` on write and +`DeserializeOwned + Zeroize` on read. This does not claim that arbitrary Serde implementations can satisfy a constant-time contract. The initial wrapper has one library-selected, versioned binary codec, not a caller-selected codec parameter. Selecting and specifying that @@ -163,7 +200,7 @@ raw bytes. Application schema evolution remains explicit: a codec version does not automatically migrate changes to `User`. **Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` -form. A matching no-context `reader.decrypt(loaded)` is proposed for symmetry; +form. A matching no-context `reader.decrypt(loaded)` is accepted for symmetry; it is not the signature of today's two-argument decrypt method. This document uses explicit context in the full example so the saved format is unambiguous. @@ -547,7 +584,7 @@ separate adapter example to add once that boundary is settled. | Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | The principal unresolved choices exposed by the examples are the opaque -ciphertext name and storage contract, the public decrypt spelling, the blob +ciphertext name and storage contract, the blob codec and timing contract, EQL context adaptation, and optional source-side output generation. None requires reintroducing public `encrypt_as`. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md new file mode 100644 index 000000000..2ede5410a --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -0,0 +1,570 @@ +# JSON tree: first-release operation contract and public examples + +Date: 2026-10-25. Companion to the +[API plan](2026-10-24-stack-encrypt-api-plan.md) and +[public workflow examples](2026-10-24-stack-encrypt-public-examples.md). + +## Status and scope + +The customer requires a comprehensive first version. This document defines +the required operation inventory for Rust and the dynamic interface used by +Go. Extraction, opening extracted values, and subtree walks are included; +they are not a post-release phase. Internal implementation can still be +split into PRs. The feature is complete only when every required row below +has Rust, dynamic/Go, and applicable PostgreSQL coverage. + +This is a target API contract, not a claim that these public signatures +already exist. Rust and JSON examples below propose the redesigned surface. +Final names/package paths are deliberately left to the last naming pass. +The [packaging glossary](2026-10-25-stack-encrypt-packaging.md#json-names-by-role) +now proposes that pass; this document retains its working names until review. +They are candidates for executable documentation during implementation; +they have not been compiled against today's API. Existing SQL spellings and +current behavior are separately identified. Do not publish these as current +SDK instructions until the acceptance suite runs them. + +Sources reviewed: + +- [Claude's Go JSON plan, PR #1353](https://github.com/cipherstash/stack/pull/1353), + head `65ece530934d367e30ea99d1f149f76117191739`, still a draft when read. + Tracking #1347; steps #1348–#1352. Its step-5 deferral is superseded here. +- [JSON tree design](2026-10-17-json-tree-index.md), including the Phase 4 + notes on extracted-path checking and stored order. +- [Engine query implementation](../../packages/stack-encrypt/src/json/query.rs), + [path grammar](../../packages/stack-encrypt/src/json/path.rs), + [derivation/opening](../../packages/stack-encrypt/src/json/derive.rs). +- [EQL target and result types](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs), + [SQL functions](../../packages/eql/src/v3/json_tree/functions.sql), + [operators](../../packages/eql/src/v3/json_tree/operators.sql), + [refused operations](../../packages/eql/src/v3/json_tree/blockers.sql), + and [leakage contract](../../packages/eql/docs/reference/json-tree-leakage.md). + +The local source baseline is the reviewed `next` commit `b1049824b` and +this documentation branch. Untracked experimental EQL files are not used +as evidence of implemented JSON-tree behavior. + +## Roles and shared policy + +| Working name | Role | +| --- | --- | +| `JsonPlaintext` | Owned JSON input/output with `Zeroize`; an adapter around the engine's JSON value model, not ciphertext or a policy | +| `JsonOptions` | Validated target options: `sealed_root` and `ordered` paths | +| `EncryptedProfile` | Application-defined fixed-policy output wrapper | +| `JsonTreeDocument` | Existing EQL encrypted document envelope, stored in one column | +| `JsonTreeQuery` | Containment/path-equality query operand; cannot decrypt | +| `JsonTreeLeafQuery` | Ordering query operand; cannot decrypt | +| `JsonTreeSelector` | Encoded path selector, not a plaintext path or ciphertext | +| `JsonTreeEntry` | Extracted encrypted node with the metadata needed to open it | +| `JsonSubtreeKey` | Client-held walk request state: token, gates, requested path and policy binding | +| `JsonTreeSubtree` | Encrypted walk rows paired with their original request state | +| `JsonSelection` | Proposed owned plaintext result distinguishing missing, one value and multiple matches; implements `Zeroize` | + +`JsonPlaintext` is a proposed library adapter, not a claim that +`serde_json::Value` implements `Zeroize`. It accepts JSON text without +requiring callers to implement their own cleanup trait. Returned plaintext +is owned by the caller. Future zeroization/timing work must not change the +stored representation. + +One immutable definition supplies options, column identity, search scope +and ciphertext binding to every operation. Rust's derive builds it from a +typed options factory; dynamic loading deserializes the same options and +validates the same constraints. Query code never supplies a second options +object that can disagree with the write policy. + +For a row-bound, tenant-scoped field: + +- Ciphertext binding: declared column + tenant scope + row ID. +- Search selectors, value terms, ordering terms and walk-token derivation: + declared column + tenant scope, with the operation's domain separation. +- Queries do not require row ID. Decrypting a selected document, entry or + subtree requires it; SELECT examples must return that ID. +- Preserve JSON node selector/ordinal authentication in addition to these + outer bindings. The engine currently combines context roles; implementing + their separation is required, not just changing a derive attribute. + +## Definitive first-release inventory + +`q` below means the typed query result, not manually assembled EQL JSON. +SQL functions shown with `eql_v4` use the existing EQL v4 names. Parameter +types are explicit to avoid PostgreSQL selecting native jsonb operators. + +| ID | Required operation | Rust surface / result | Dynamic operation / result | Database use | +| --- | --- | --- | --- | --- | +| J01 | Encrypt document: object, array, scalar or JSON null | `value.encrypt_into` → encrypted target | `encrypt` → typed EQL document | Store `eql_v4_json_tree` | +| J02 | Decrypt complete document | `JsonPlaintext::decrypt_from` → JSON | `decrypt` → JSON text | Read full column | +| J03 | Batch write/read | `Vec<_>` conversion composition | `encrypt_many`, `decrypt_many` | Preserve row/result ordering; shared executor | +| J04 | Contains | `query(op::Contains, value)` → `JsonTreeQuery` | `contains` → same EQL operand | `profile @> $1::eql_v4.query_json_tree` | +| J05 | Contained by | `query(op::ContainedBy, value)` → `JsonTreeQuery` | `contained_by` | `profile <@ $1::eql_v4.query_json_tree` | +| J06 | Scalar equality at path | `query(op::AtPath(op::Eq), (path, scalar))` → `JsonTreeQuery` | `eq_at` | Same containment predicate as J04 | +| J07 | Four ordering predicates | `AtPath(Lt/Lte/Gt/Gte)` → `JsonTreeLeafQuery` | `lt_at`, `lte_at`, `gt_at`, `gte_at` | `(profile -> $1::text) < $2::eql_v4.query_json_tree_leaf`, etc. | +| J08 | Inclusive range | `AtPath(Between)` → two leaf operands | `between_at` → lower/upper operands | `>= lower AND <= upper` | +| J09 | Sort ascending/descending | `Selector` query → selector; no bound value | `selector` | `ORDER BY eql_v4.ord_term(profile -> $1::text) ASC/DESC` | +| J10 | Path exists | `Selector` query | `selector` | `json_tree_path_exists(profile::jsonb, $1::text)` | +| J11 | Extract first match | `Selector` query → optional encrypted entry from DB | `selector`; result accepted by `decrypt_entry` | `profile -> $1::text` or `json_tree_path_query_first` | +| J12 | Extract first match as text | Same selector; parse text into `JsonTreeEntry` | Same selector; parse encrypted entry text | `profile ->> $1::text`; never plaintext | +| J13 | Extract all matches | Same selector → encrypted entry rows | Same selector; `decrypt_entries` | `json_tree_path_query(profile::jsonb, $1::text)` | +| J14 | Open extracted scalar(s) | `decrypt_from` on source bound to selection request | `decrypt_entry`, `decrypt_entries` with selection handle | No server decrypt; container markers refused | +| J15 | Prepare subtree/positional request | `query(op::Subtree, path)` → `JsonSubtreeKey` | `subtree_key` → handle + SQL parameters | Token/gates, derived under field scope/options | +| J16 | Fetch/open subtree | Request-bound `JsonTreeSubtree`, then `decrypt_from` | `decrypt_subtree` with walk handle | `json_tree_subtree` / `json_tree_subtree_at` | +| J17 | Enumerate immediate children | J15 key, encrypted child rows | J15 key; opening scalar children uses J14 with walk validation | `json_tree_children(profile, token, gates)` | +| J18 | Count immediate children | J15 key; result is integer | J15 key; database returns integer | `json_tree_child_count(profile, token, gates)` | +| J19 | Walk budget and sealed-root fallback | Same J16 plaintext result regardless of path taken | Same handle and result shape | Default budget 64; zero walks without fallback | +| J20 | Compare stored documents/ordered entries within one scope | Existing typed EQL operands; no new plaintext term | Existing typed EQL operands | Document `@>`/`<@`; entry `< <= > >=`; same policy/scope required | + +J09–J13 and J17–J18 are SQL actions using a derived selector/key, not new +cryptographic primitives or SDK database execution methods. The dynamic +engine derives operands and opens results; the caller runs SQL. The +definition's capability discovery must describe both operand preparation +and the supported database use, including ordering-path restrictions. + +J20 does not authorize whole-document `=` or equality via an ordering +term. Initial ORE support never answers `Eq`. JSON scalar equality remains +available through its independent exact value-selector mechanism. Any +future ORE equality capability must be explicit and limited to encodings +that preserve equality; text/lossy encodings remain ineligible. + +## Path, null and result semantics + +- Grammar: `$`, `.member`, `["member"]`, nonnegative `[n]`, and `[*]`. + No filters, recursive descent or slices in this release. +- Selector-based operations address a path class across array elements. + Existing selectors erase positions (`[2]` has the same selector as + `[*]`). **Proposed first-class API rule:** reject positional paths for + `EqAt`, selector extraction/existence and selector ordering rather than + silently broadening them. Use the positional subtree route for exact + `[n]` retrieval. This is a deliberate stricter wrapper than today's engine. +- Wildcard equality means at least one matching element has that scalar + value. Containment preserves grouping within array elements; it is not + an ungrouped bag of independent terms. Multiple independent EqAt SQL + predicates do not imply the values belong to the same array element. + Use one containment needle to require that correlation. +- Equality accepts null, boolean, string and supported JSON number values; + objects/arrays are refused. Containers use containment, not an operator + labelled equality. At `$`, equality asks for that scalar root, not an + array containing it (which is containment semantics). +- Ordering is opt-in per path and accepts supported numbers/strings only. + No implicit ordering at undeclared paths. A wildcard ordering policy can + supply terms to many entries: `->` compares/sorts the first matching + entry, while `json_tree_path_query` can enumerate all and an explicit + SQL `EXISTS` chooses any-match semantics. Never present first-match as + any-match, or term equality as exact plaintext equality. +- A missing path gives no extraction row (or SQL NULL for first-match). + A present JSON null gives an encrypted entry that opens to JSON null. + SQL NULL for an absent document is a separate field-null policy, not an + encrypted JSON null. An empty object/array is neither missing nor null. +- JSON inputs cross the dynamic ABI as UTF-8 JSON text. No intermediate + conversion of JSON numbers to Go `float64`; use raw JSON/number-preserving + decoding. Define accepted numeric range and canonicalization from the + engine, test `2`/`2.0`, negative zero and large integers. Reject values + outside the supported representation rather than silently rounding them. +- SQL returns encrypted data, even with `->>`. A scalar node opens directly; + an ordinary container node is a marker, not a serialized subtree. Use a + subtree walk to reconstruct containers. The sealed root holds the whole + document and has an explicit whole-document/fallback decode path. +- Selection/open requests retain expected selector, field definition and + scope. Check the returned selector before opening: current code opens + under server-supplied selectors, which is insufficient to establish the + requested path. Row provenance requires explicit row binding. Walks must + validate the path/selector relationships needed for their requested result. + Successful decryption does not prove query completeness or authenticate + every search/index metadata field. +- Requests for `$` (or root-array wildcards requiring a root token) under + `sealed_root=true` use whole-document fetch/decrypt and local selection; + never fabricate a nonexistent root token. For a missing/empty/scalar + subtree where a descendants-only walk cannot distinguish the cases, use + the selected start-node metadata or full-document fallback explicitly; + do not treat an empty row set as an empty object by assumption. + +## Rust write/read example + +Proposed public code. `target(column = ...)` is illustrative declaration +syntax for an explicit EQL column identity; the exact attribute spelling +belongs to the forthcoming specification/naming pass. The options factory +is typed Rust, not JSON hidden inside an attribute. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use serde::{Deserialize, Serialize}; +use stack_encrypt::{DecryptFrom, DecryptSource, EncryptInto, EncryptTarget, StackCipher}; +use stack_encrypt::json::{JsonOptions, JsonPlaintext}; +use stack_encrypt::sem::TermError; +use std::error::Error; +use zeroize::Zeroize; + +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].date"]) +} + +#[derive(Serialize, Deserialize, EncryptTarget, DecryptSource)] +#[serde(transparent)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("users", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); + +#[tokio::main] +async fn main() -> Result<(), Box> { + let client = StackCipher::new().await?; + let keyset = client.default_keyset(); + let _definition = EncryptedProfile::definition()?; + + let profile = JsonPlaintext::parse( + r#"{"age":42,"name":"Alice","visits":[{"date":"2026-10-01"}]}"#, + )?; + let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; + + // These bytes can instead be bound to the EQL PostgreSQL column. + let stored = serde_json::to_vec(&encrypted)?; + let loaded: EncryptedProfile = serde_json::from_slice(&stored)?; + let mut opened = JsonPlaintext::decrypt_from(loaded, &keyset).await?; + // The caller now owns the returned plaintext and its cleanup. + opened.zeroize(); + + let profiles = vec![ + JsonPlaintext::parse(r#"{"age":42}"#)?, + JsonPlaintext::parse(r#"{"age":27}"#)?, + ]; + let encrypted: Vec = profiles.encrypt_into(&keyset).await?; + let mut opened = Vec::::decrypt_from(encrypted, &keyset).await?; + assert_eq!(opened.len(), 2); + opened.zeroize(); + Ok(()) +} +``` + +This example explicitly declares column-wide scope and no row binding by +its fixed standalone definition; the normative schema must express that +choice rather than silently replacing missing required tenant scope. +For an output-derived `EncryptedUser` with `id: PrimaryKey` and +`tenant_id: ScopeKey`, select those fields through the context/scope +attributes described in the API plan. Obtain its JSON field handle from +`EncryptedUser::definition()?.profile()` and require `TenantId` on query +requests. Store/return both ID fields for decrypt. If caller input is used, +use the agreed `_with_context` variants with a typed input separating +binding and scope. No row ID is ever an index/query input implicitly. + +## Rust query and extraction recipes + +These snippets share `keyset` and a resolved definition `profile` from the +write example. Add `use stack_encrypt::{op, json::JsonPath};`. The `using` +selector chooses that definition before the operation. `Selector` and +`Subtree` are proposed operation markers integrating today's separate +`json_selector`/`json_subtree_key` helpers into the shared definition API. +All query operands below use the same plaintext JSON adapter as writes. + +```rust +let profile = EncryptedProfile::definition()?; +let contains = keyset.using(&profile) + .query(op::Contains, JsonPlaintext::parse(r#"{"name":"Alice"}"#)?) + .await?; +let contained_by = keyset.using(&profile) + .query(op::ContainedBy, JsonPlaintext::parse(r#"{"age":42,"name":"Alice"}"#)?) + .await?; +let eq = keyset.using(&profile) + .query(op::AtPath(op::Eq), ("$.name", JsonPlaintext::parse(r#""Alice""#)?)) + .await?; + +let lt = keyset.using(&profile) + .query(op::AtPath(op::Lt), ("$.age", JsonPlaintext::parse("65")?)).await?; +let lte = keyset.using(&profile) + .query(op::AtPath(op::Lte), ("$.age", JsonPlaintext::parse("65")?)).await?; +let gt = keyset.using(&profile) + .query(op::AtPath(op::Gt), ("$.age", JsonPlaintext::parse("18")?)).await?; +let gte = keyset.using(&profile) + .query(op::AtPath(op::Gte), ("$.age", JsonPlaintext::parse("18")?)).await?; +let range = keyset.using(&profile) + .query(op::AtPath(op::Between), ( + "$.age", (JsonPlaintext::parse("18")?, JsonPlaintext::parse("65")?), + )).await?; + +let age = keyset.using(&profile) + .query(op::Selector, JsonPath::parse("$.age")?).await?; +let visit_dates = keyset.using(&profile) + .query(op::Selector, JsonPath::parse("$.visits[*].date")?).await?; +``` + +The results carry named typed EQL payloads. For example, `lt.selector()` +and `lt` bind the selector and leaf operand in the ordering SQL below; +`range.lower()`/`upper()` name the two bounds (proposed accessors). +For static policies, generated ordered-path handles can add compile-time +checks; runtime paths still validate against the same definition before +term derivation. Missing definition/scope makes a Rust request unawaitable. + +```sql +-- Containment/equality operands: bind the named query payload, not plaintext. +SELECT id, tenant_id, profile FROM users +WHERE profile @> $1::eql_v4.query_json_tree; + +SELECT id, tenant_id, profile FROM users +WHERE profile <@ $1::eql_v4.query_json_tree; + +-- Lower and upper bounds; use <, <=, >, >= individually for J07. +SELECT id, tenant_id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_tree_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_tree_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Existence, first encrypted entry, first entry as encrypted JSON text. +SELECT id, tenant_id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, tenant_id, profile -> $1::text AS entry FROM users; +SELECT id, tenant_id, profile ->> $1::text AS entry_text FROM users; + +-- All matching entries: group results by document before opening them. +SELECT u.id, u.tenant_id, e.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS e(entry); + +-- Stored-to-stored comparisons require matching policy and search scope. +SELECT a.id, b.id FROM users AS a JOIN users AS b + ON a.tenant_id = b.tenant_id AND a.profile @> b.profile; +SELECT a.id, b.id FROM users AS a JOIN users AS b + ON a.tenant_id = b.tenant_id + AND (a.profile -> $1::text) < (b.profile -> $1::text); +``` + +Add the application's tenant filter where appropriate; cryptographic scope +does not replace SQL authorization. A selector identifies a path class; +the SQL above never receives `$.age` or another plaintext JSON path. + +Proposed request-bound opening preserves the expected path and avoids the +current server-supplied-selector ambiguity. Here `entry` and `entries` are +typed database results for the corresponding selection request, and each +batch belongs to the identified document/context: + +```rust +let value = JsonPlaintext::decrypt_from(age.bind(entry)?, &keyset).await?; +let values = Vec::::decrypt_from( + visit_dates.bind_many(entries)?, &keyset, +).await?; +``` + +`bind` is a proposed checked source adapter, not a second decrypt method. +For row-bound policies, call `decrypt_from_with_context` with that result +row's expected ID and required scope. Missing first-match rows stay absent; +do not manufacture an encrypted null. Containers use the walk route below. + +## Walks, children, positional retrieval and fallback + +```rust +let walk = keyset.using(&profile) + .query(op::Subtree, JsonPath::parse("$.visits[0]")?).await?; + +// After the SQL below returns all rows for one document: +let source = walk.bind_subtree(rows)?; +let visits = JsonSelection::decrypt_from(source, &keyset).await?; +``` + +`walk` retains the requested path, resolved scope and definition; only the +token/gates go to SQL. Exact positions are supported here, including the +necessary selector gate to distinguish an array element from an object +member at the same ordinal. The final specification must normalize result +cardinality through `JsonSelection`: a concrete path yields `Missing` or +`One(JsonPlaintext)`; wildcard paths yield `Many(Vec)` in +document order. Missing and an explicit JSON null remain distinguishable. +`bind_subtree`/opening must preserve that distinction. `JsonSelection` is a +working type name and is imported from the proposed JSON module. + +```sql +-- $1: token; $2: gates; $3: budget (64 default, 0 disables fallback). +SELECT u.id, u.tenant_id, w.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree_at( + u.profile, $1::text, $2::text[], $3::integer +) AS w(entry); + +-- With no gates, the dedicated form is also supported. +SELECT u.id, u.tenant_id, w.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text, $2::integer +) AS w(entry); + +-- Immediate children and counts use the same token/gates. +SELECT u.id, u.tenant_id, c.entry +FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children( + u.profile, $1::text, $2::text[] +) AS c(entry); +SELECT id, tenant_id, + eql_v4.json_tree_child_count(profile, $1::text, $2::text[]) +FROM users; +``` + +Use one request-bound source per document; do not merge entries from several +rows into a subtree. Immediate scalar children can use checked entry +opening; container children need their complete descendants or an explicit +full-document fallback. The SDK must expose this distinction rather than +pretend that a container marker decrypts to its children. + +Budget 64/0 are current SQL semantics to retain in the first public scope. +When `sealed_root=true`, an over-budget walk returns the sealed root instead +of partial data; the client opens it and selects the original requested +path. With `sealed_root=false`, walk the full result or report an explicit +resource limit, never silently truncate it. The same requested value(s) +must result from budget 0 and fallback. Tokens reveal subtree structure +within their search scope and are not decryption keys; document their +existing leakage and lifetime with these examples. + +## Dynamic definition and operation examples + +These are logical JSON messages for the specification, not a promise to +replace the frame ABI with JSON. The host/guest transport encodes their +typed equivalents. `using` identifies a validated field definition; the +engine, not Go, creates EQL envelopes and query terms. Naming the definition +in a request must resolve to the same options and scope contract as writes. + +The fixed-column example's dynamic counterpart is: + +```json +{ + "version": 1, + "name": "users.profile", + "plaintext": "json", + "target": { + "name": "JsonTreeDocument", + "column": { "table": "users", "column": "profile" }, + "options": { "sealed_root": true, "ordered": ["$.age", "$.visits[*].date"] } + }, + "search_scope": { "kind": "column" }, + "context": { "kind": "none" } +} +``` + +`version: 1` is an illustrative schema version, to allocate in the formal +specification. Named types are optional. For tenant/row policies use the +accepted `ScopeKey`/`PrimaryKey` declarations and source selectors; the +resolved query input requires the tenant scope but never the row ID. + +Request corpus (each array element is an independent example, not one +batch of mixed operations). JSON text is deliberately explicit so no host +float conversion precedes the engine parser: + +```json +[ + { "operation": "encrypt", "using": "users.profile", "value": { "json_text": "{\"age\":42}" } }, + { "operation": "encrypt_many", "using": "users.profile", "values": [{ "json_text": "{\"age\":42}" }, { "json_text": "{\"age\":27}" }] }, + { "operation": "query", "using": "users.profile", "operator": "contains", "input": { "json_text": "{\"age\":42}" } }, + { "operation": "query", "using": "users.profile", "operator": "contained_by", "input": { "json_text": "{\"age\":42,\"name\":\"Alice\"}" } }, + { "operation": "query", "using": "users.profile", "operator": "eq_at", "input": { "path": "$.name", "json_text": "\"Alice\"" } }, + { "operation": "query", "using": "users.profile", "operator": "lt_at", "input": { "path": "$.age", "json_text": "65" } }, + { "operation": "query", "using": "users.profile", "operator": "lte_at", "input": { "path": "$.age", "json_text": "65" } }, + { "operation": "query", "using": "users.profile", "operator": "gt_at", "input": { "path": "$.age", "json_text": "18" } }, + { "operation": "query", "using": "users.profile", "operator": "gte_at", "input": { "path": "$.age", "json_text": "18" } }, + { "operation": "query", "using": "users.profile", "operator": "between_at", "input": { "path": "$.age", "lower": { "json_text": "18" }, "upper": { "json_text": "65" } } }, + { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.age" } }, + { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.visits[*].date" } }, + { "operation": "query", "using": "users.profile", "operator": "subtree_key", "input": { "path": "$.visits[0]" } } +] +``` + +Responses carry a result kind and typed payload: document, containment +operand, leaf operand, range, selection, or walk request. SQL preparation +returns the named domain and encoded payload (or typed token/selector +parameters); it does not concatenate SQL. Selectors serve J09–J13, and walk +keys serve J16–J19, so those SQL uses do not need redundant engine opcodes. + +For decode operations, this table specifies the dynamic signatures. Values +such as `document` and `entries` mean complete validated EQL payloads, not +literal placeholder strings to parse as ciphertext: + +| Operation | Required inputs | Result | +| --- | --- | --- | +| `decrypt` | `using`, `document`, required context/scope | JSON text | +| `decrypt_many` | `using`, documents with per-record context/scope | JSON texts in input order | +| `decrypt_entry` | `using`, `selection_handle`, entry, required context/scope | Scalar JSON text | +| `decrypt_entries` | `using`, selection/walk handle, entries grouped by record and context | Ordered scalar JSON texts | +| `decrypt_subtree` | `using`, `walk_handle`, complete rows for one document, required context/scope | Explicit missing / one JSON value / multiple matches | + +The response to `selector`/`subtree_key` includes an opaque request handle +in addition to SQL parameters. The handle retains the path and resolved +definition/scope in the engine, so callers cannot accidentally substitute +arbitrary path/options during decode. Specify handle lifetime and release +in the guest ABI; handles are not persistent database values. If the host +must recreate one after restart, derive it again from the original request, +not from server-returned metadata. Stale or mismatched handles are errors. +An equivalent Rust source adapter retains the same state directly. + +Example host orchestration, using proposed dynamic method names rather +than inventing ciphertext strings in a JSON example: + +```rust +let loaded = dynamic.load_definition(definition_json)?; +let encrypted = loaded.encrypt(json_text, inputs).await?; +let stored = database.store(encrypted).await?; +let opened = loaded.decrypt(stored, decrypt_inputs).await?; + +let selection = loaded.selector("$.age", scope).await?; +let entry = database.select_first(selection.sql_parameters()).await?; +if let Some(entry) = entry { + let age = loaded.decrypt_entry(&selection, entry, row_inputs).await?; +} + +let walk = loaded.subtree_key("$.visits[0]", scope).await?; +let rows = database.subtree(walk.sql_parameters(), 64).await?; +let result = loaded.decrypt_subtree(&walk, rows, row_inputs).await?; +``` + +This is an orchestration recipe: `database` represents application SQL, +not a new Stack Encrypt database driver; the input variables are supplied +by that application. Generated Go methods should hide these message details, +use named results, and call the same validated dynamic operations. JSON +marshalling helpers must not force a containment needle into a whole record +struct whose zero-valued fields accidentally narrow the query. + +## Explicitly unsupported operations + +The comprehensive first release covers the inventory, not all PostgreSQL +JSON operators. Reject whole-document `=`/`<>` as plaintext equality, +entry equality through ordering terms, plaintext path SQL operands, +JSONPath filters/recursive descent/slices, and in-place encrypted document +patch/delete/concatenation. Updates read/decrypt, modify, then encrypt the +complete document. Projection extraction does not confer the ability to +write a standalone node back into an existing tree. + +Reject malformed JSON/path/options; unsupported numeric values; container +EqAt operands; boolean/null ordering operands; undeclared ordering paths; +selector-position ambiguity; missing scope/context; unknown/expired request +handles; and wrong definition/scope/path during opening. Preserve useful +reason codes and structural locations without echoing plaintext values. +Structural validation that needs no key must precede key I/O. The current +engine refusal names include `JsonValueNotScalar`, `JsonLeafNotOrdered`, +`JsonPathNotOrdered`, `JsonOrderingAtPosition`, `JsonTooDeep`, +`JsonTooWide`, `JsonSealedRoot`, and `OpenedType`; final dynamic codes must +map deliberately rather than exposing unstable Rust enum spellings. + +## Implementation gaps and release gates + +1. **Shared definitions and scope separation.** Implement fixed/dynamic + target resolution, typed options, scope-role validation, row-binding + inputs and matching write/query context derivation. Generate capability + metadata from one registry, including JSON-specific operand/result kinds. +2. **Catalog/guest coverage.** Complete the JSON tree catalog, target table, + guest registration, dynamic index execution and JSON-text record kind. + Current Rust helpers/EQL types are not evidence that Go supports them. +3. **Direct EQL adapters.** Replace the existing serialization round trip + with typed conversions for documents, entries and operands. +4. **Selectors and opening.** Add the request-bound source adapters and + dynamic handles, validate expected selectors, preserve required row + context, and test mismatched results and missing headers/ordinals. +5. **Walk completeness.** Cover children/counts, concrete positions, + wildcards, empty containers, missing paths, scalar starts, root requests, + sealed-root on/off and budgets 0/64. Resolve reconstruction ambiguities + explicitly, including returning/fetching the start node when needed. + Do not ship opening/walks later under the earlier PR #1353 scope. +6. **Conformance/documentation.** Turn J01–J20 into identified test cases + and the examples into checked docs. Run Rust writes → Go reads/queries + and Go writes → Rust reads/queries against PostgreSQL. Include negative + cases, exact number handling, null distinctions, multi-element correlated + containment, and batch ordering. Test SQL GIN/btree usage on representative + containment/range queries. Randomized ciphertexts need semantic parity; + deterministic fixtures are only for explicitly seeded test builds. + +The current wire order/leakage and extraction trust boundaries must be +documented accurately. This operation inventory does not claim a new +authenticated-query-completeness protocol or a storage-order leakage fix. +Stable blob framing is a separate deliverable tracked in +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); it must +not be confused with the EQL JSON-tree storage format. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md new file mode 100644 index 000000000..7b4e749d3 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -0,0 +1,253 @@ +# Stack Encrypt packaging and public names + +Date: 2026-10-25. Recommendation for review, accompanying the +[implementation plan](2026-10-24-stack-encrypt-api-plan.md). +These are proposed boundaries and names, not implemented APIs or accepted +renames. Existing companion examples retain their working names until this +proposal is accepted. The detailed JSON operation contract remains under +separate user review. + +Source baseline: the current documentation branch and inspected local Vitamin +C 0.8.0 sources. Stack's +[cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) +explain its existing batching boundary. The +[EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) establishes +the optional downstream engine dependency, and +[encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) +are the concrete shared-type extraction candidates. This is a source-level +design assessment, not an audit of every external Vitamin C consumer. + +## Ownership rule + +Stack Encrypt owns the application value model and its execution: turning +owned Rust/dynamic values into ciphertext, structured targets and search +operands, and reconstructing plaintext. Vitamin C owns reusable cryptographic +and protected-memory primitives. A normal Stack Encrypt user should not need +to import Vitamin C encryption traits to implement an application type. + +Moving code should remove duplicate abstractions, not preserve both public +surfaces under new names. The user's prototype compatibility exemption +applies to this API; review other Vitamin C consumers before deleting their +dependencies upstream. + +## Proposed packages + +| Package | Responsibility | +| --- | --- | +| `stack-encrypt` | Public Rust API, value traversal/reconstruction, target declarations and definitions, context/scope roles, normalization and searchable-target adapters, plans, batching, JSON-tree execution, and dynamic definition validation/execution | +| `stack-encrypt-derive` | Derives generating the same declarations and definitions available by hand; re-export derives from `stack-encrypt` | +| `stack-encrypt-format` — new | Pure data representations/codecs shared by the engine and bindings: frame primitives and required scalar representations, stable blob profile, ciphertext envelope parsing/encoding, and validated Base64/Base85 text; no key requests or execution runtime | +| `stack-zerokms` — rename current `stack-kms` | ZeroKMS protocol client, authentication integration, native keysets, lock-context request types, and responses; it is not the neutral multi-provider abstraction | +| `eql-bindings` | EQL versioned storage/query types, SQL-compatible JSON serialization, catalog and Stack target adapters; stays downstream of the engine | +| Vitamin C crates | Protected memory, randomness, byte-level AEAD/key operations, HMAC/PRF and ORE algorithm primitives, and reusable canonical context encoding | + +Keep `stack-auth` and `stack-profile` separate: they have consumers outside +encryption. Keep `stack-guest-abi` for the host/guest calling convention. +Neither the ABI nor the Go host should own a second target-policy model. +Put the formal dynamic contract, schemas and conformance fixtures beside +Stack Encrypt; its implementation uses the same definition types and +validation as static Rust. + +Use modules for the remaining boundaries initially. In particular, do not +create separate execution, JSON-policy, query, context or provider-traits +crates merely to mirror each concept. + +The format crate has a concrete consumer justification: EQL's wire types +currently work without its optional `stack-encrypt` integration. They need +shared encoding types without an encryption client/runtime dependency. Make +Serde, schema and TypeScript support feature-scoped where practical; define +implementations alongside the owned types so downstream orphan rules do not +force duplicate wrappers. Re-export the ordinary format types through Stack +Encrypt, so users normally need only its dependency. + +The format crate is a single implementation with distinct profiles. Blob +storage, host/guest transport, and ciphertext envelopes have different +contracts and versioning requirements; sharing code must not silently make +the ephemeral transport Serde mapping the persisted blob format. This does +not introduce persistence for arbitrary internal `Node` trees: structured +outputs still use their explicit target layout and ordinary Serde composition. + +## Dependency direction + +```mermaid +flowchart TD + App[Application / Rust or guest adapter] --> EQL[eql-bindings] + App --> Engine[stack-encrypt] + EQL -->|optional target integration| Engine + EQL --> Format[stack-encrypt-format] + Engine --> Format + Engine --> Derive[stack-encrypt-derive] + Engine -->|optional ZeroKMS adapter| KMS[stack-zerokms] + Engine --> Vitamin[Vitamin C primitives] +``` + +The derive emits Stack paths but does not depend on the engine as a normal +runtime dependency. No Vitamin C primitive depends back on Stack Encrypt. +`stack-zerokms` does not depend on the engine. Provider traits belong to +`stack_encrypt::provider`; a Stack-owned adapter implements them using the +ZeroKMS client. This avoids an engine/client dependency cycle. + +Make ZeroKMS an optional provider feature independently of the HTTP feature, +so a different provider does not pull its protocol into the core contract. +Preserve injected providers and guest/native builds. Data-key support is the +base capability; index-key support, named-keyset resolution and lock context +remain explicit optional capabilities. Unsupported required capabilities +fail before key I/O. Future provider adapters can become separate downstream +crates when needed; do not design fictitious AWS/Vault keysets now. + +Do not re-export EQL from Stack Encrypt: EQL's target integration already +depends on Stack Encrypt. Applications explicitly import their EQL types. + +## What moves out of Vitamin C + +Move application-facing plaintext/ciphertext visitors, structural nodes, +value conversion traits and derives, tree walking, collection composition, +and application encryption helpers into Stack's value/target/plan layers. +Keep advanced extension points public only where target implementers need +them; they are not the normal root API. Internal nodes do not become a +second public opaque ciphertext type. + +Move the generic frame machinery and codecs needed by Stack into the format +crate. Stack owns the stored blob profile and its compatibility promise. +This refines the earlier recommendation to reuse Vitamin C's frame codec: +reuse its implementation, with a lossless storage mapping, while changing +its long-term ownership. Coordinate this handoff under +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); retain +the issue's format specification and fixture work. No upstream issue or +repository has been changed by this proposal. + +Also move normalization and PRF/ORE adapters that understand Stack's logical +values, traversal, targets or query semantics. Keep independent cryptographic +algorithms below them. The current `vitaminc-normalize`, `vitaminc-prf` and +`vitaminc-ore` manifests depend on `vitaminc-aead`; normalization even uses +its physical scalar types. Moving only the encryption traits would therefore +leave a dependency back to the old value model. Separate primitive inputs +from application adapters as part of the move; do not add a reverse +Vitamin C dependency on Stack to make it compile. + +Retain canonical context byte construction in `vitaminc-context` where it +serves independent algorithms. Stack owns the semantic roles and selection +rules (`PrimaryKey`, `ScopeKey`, ciphertext context, query scope and caller +input capabilities), reusing that canonical encoding instead of inventing +an incompatible second one. Provider policy remains a separate role. + +## Retire AsyncCipher + +Recommend removing Vitamin C's `AsyncCipher` and its associated application +execution helpers as the Stack executor takes ownership. Do not introduce a +replacement public catch-all async cipher trait in Stack. + +Stack already plans and groups remote key operations, then performs local +synchronous cryptography. Its current `MintedKeys`/`BoundKeys` implementations +use synchronous leaf cipher operations; `AsyncCipher` models a different +place to put asynchronous work. The meaningful async boundary is the provider +and the public pending operation, with batching controlled by Stack. + +It is unused by Stack, but not unused inside Vitamin C: `aead/src/seal.rs`, +`context.rs`, `typed.rs`, `walk.rs` and `normalize/src/terms.rs` expose or use +async helpers, and their tests implement it. Migrate the relevant batching, +ordering, failure and reconstruction coverage with the code, then delete +the retired trait/helper surface. Audit external consumers before the +upstream release. Separate internal leaf seal/open capabilities where useful +instead of retaining implementations that reject half of a combined trait. + +## Recommended public glossary + +| Name/path | Meaning | +| --- | --- | +| `stack_encrypt::Cipher` | Provider-backed execution handle; replaces `StackCipher` | +| `KeysetCipher` | Execution handle bound to a resolved keyset, available for providers supporting keysets | +| `Ciphertext` | One opaque encrypted value with a specified storage envelope; replaces the public use of `StackCipherText = Node` | +| `OpaqueBlob` | Explicit owned whole-value serialization adapter | +| `EncryptInto`, `DecryptFrom` | Target-directed encryption and destination-directed decryption convenience traits, with context variants | +| `OpaqueEncode`, `OpaqueDecode` | Single-ciphertext representation traits | +| `target::{EncryptTarget, DecryptSource}` | Declaration traits; matching derives are re-exported at the crate root | +| `target::TargetDefinition` | Validated immutable definition shared by static and dynamic paths | +| `context::{PrimaryKey, ScopeKey}` | Eligibility wrappers; neither automatically selects a context or keyset | +| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes; owned in the format crate and re-exported here | +| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated encoded opaque-ciphertext representations and supported field targets; encoding validity alone is insufficient | +| `plan::Plan` | Explicit low-level composition interface; output selection is `.target::()` | +| `provider` | Neutral provider contracts and capability bounds | +| `provider::zerokms` | ZeroKMS adapter and policy extension surface, including `LockContext` | +| `dynamic` | Versioned definition loading, capability discovery and execution through the common engine | + +Keep the already-agreed `.using(...)`, `definition()`, `encrypt`, +`encrypt_with_context`, `encrypt_into`, `encrypt_into_with_context`, +`decrypt_from` and `decrypt_from_with_context` spellings. Keep direct +`cipher.decrypt`/`decrypt_with_context`. Remove plaintext `.encrypt`, +redundant `encrypt_as`, and public `decrypt_into` as already agreed. + +Use `TargetDefinition` for the configured public contract and `Layout` only +for the advanced internal composition concept where that distinction is +needed. A layout does not name a plaintext document or a stored ciphertext. +Name low-level operations `seal`/`open` where they act on bytes; reserve the +application encrypt/decrypt vocabulary for the public value operations. + +Keep ciphertext representations in the format crate independent of traits; +Stack implements its own target traits for those types. Plain encoded text +is not itself an encryption target. Construction must validate both layers +for the ciphertext wrappers; avoid an unchecked public string constructor. + +## JSON names by role + +Keep the existing EQL package and version modules. Avoid importing low-level +engine JSON documents into normal application examples alongside EQL output +types. Recommend this glossary for the first-class public surface: + +| Working/current name | Recommended name | Role / home | +| --- | --- | --- | +| `JsonPlaintext` | `JsonPlaintext` | Owned plaintext adapter, `stack_encrypt::json` | +| `JsonOptions` | `JsonTreeOptions` | Validated tree-target policy, `stack_encrypt::json` | +| `JsonSelection` | `JsonSelection` | Proposed plaintext selection result, `stack_encrypt::json` | +| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document, `eql_bindings::v4` | +| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry, `eql_bindings::v4` | +| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result, `eql_bindings::v4` | +| `JsonTreeQuery` | `JsonContainmentQuery` | Containment/path-value query operand, `eql_bindings::v4` | +| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Ordering query operand, `eql_bindings::v4` | +| `JsonTreeSelector` | `JsonSelector` | Encoded path selector, `eql_bindings::v4` | +| `JsonSubtreeKey` | `JsonWalkRequest` | Client-held walk token and request binding; engine type with a typed EQL adapter | + +`JsonContainmentQuery` can still be the physical operand for a path equality +operation where the agreed SQL recipe uses containment; operation and wire +operand need not have the same name. `JsonWalkRequest` describes the actual +role better than "key": it carries sensitive capability/request state, +not the data-encryption key. Exact operation behavior remains governed by +the separate JSON contract review. + +Rename the internal target index `Json` to `JsonTreeIndex`. Keep the internal +engine `JsonDocument` out of the ordinary facade, with a role-qualified name +such as `EncryptedJsonTree` if adapter authors require public access. +`EncryptedJson` always denotes the EQL-compatible stored form, not any +arbitrary JSON serialization of encrypted nodes. An application's +`EncryptedUser` remains its own declared record layout. + +Dynamic target/operation identifiers are explicitly versioned contract +identifiers, not Rust type names serialized into JSON. Select their exact +spellings in the interoperability specification and use the same glossary +in Go. Rust renames alone must not accidentally redefine stored SQL formats +or dynamic discriminants. + +## Implementation order and acceptance + +1. Approve this glossary and ownership map; update all public example imports + and type names together without implying the examples already compile. +2. Specify the stable format profiles and conformance fixtures, coordinating + #519. Establish the small format crate and move the encoding wrappers and + frame foundation; preserve distinct transport/storage semantics. +3. Move the value model, traversal and dependent normalization/index adapters + into Stack, separating primitive inputs so the dependency graph stays + acyclic. Move relevant tests with their implementations. +4. Introduce neutral provider capabilities and the ZeroKMS adapter, rename + the protocol crate, then retire Vitamin C's async application execution + surface. Keep mixed-keyset/auth-policy batching and request isolation. +5. Apply the agreed method/derive/JSON names across Rust, EQL, dynamic catalog, + guest ABI adapters, Go generation and examples. Preserve the comprehensive + first-release JSON inventory rather than treating packaging as a scope cut. + +Acceptance includes an acyclic Cargo graph; an EQL wire-only build without +the engine; engine builds without ZeroKMS and without HTTP; native/guest +builds; compiled public examples; the moved behavior tests; and shared +Rust/Go/SQL format/operation fixtures. Full timing and memory-cleanup +improvements remain follow-up work under the accepted format-first priority; +record actual guarantees honestly. No implementation changes, tests of the +proposed API, code moves, or upstream removals are claimed by this document. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md new file mode 100644 index 000000000..eeaf0afef --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -0,0 +1,342 @@ +# Key selection, lock context and provider capabilities + +Date: 2026-10-25. Assessment accompanying the +[API implementation plan](2026-10-24-stack-encrypt-api-plan.md). + +Status: recommendations for review, not accepted public signatures or an +implementation. The user requested plaintext-derived ZeroKMS keyset +selection, review of Vitamin C `AsyncCipher`, and optional lock-context +support without requiring future AWS KMS/Vault backends to emulate it. +The JSON operation contract remains under user review independently. + +## Distinct roles + +| Role | Selects/binds | Example | +| --- | --- | --- | +| Key selection | Provider key domain used to encrypt and derive index terms | ZeroKMS keyset ID or name selected from `tenant_id` | +| Ciphertext context | Application binding verified when opening ciphertext | Column + tenant scope + row ID | +| Search scope | Domain in which index/query terms compare | Column + tenant ID | +| Provider policy | Conditions the provider applies to key generation/retrieval | ZeroKMS lock context selecting authenticated JWT `sub` | + +These are separate logical inputs, even where the protocol binds several +into its tag/AAD. A `ScopeKey` does not select a ZeroKMS keyset by itself; +a keyset selector does not insert its source field into AAD; lock context +is not an alias for either ciphertext context or search scope. Sharing a +source value requires explicit declaration, not inference from a name. + +## Current implementation evidence + +- [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts + `IdentifiedBy` by UUID or name, resolves the keyset and its index key, + and returns a bound `KeysetCipher`. The client has a bounded keyset cache + and name-resolution freshness policy. `.default_keyset()` is already + bound at initialization. Current initialization requires both + `DataKeySource` and `IndexKeySource`. +- A sealed leaf carries the resolved keyset UUID. Client-scoped decrypt + groups retrieval by the UUIDs in the leaves; a bound keyset can constrain + which keyset is permitted. Names are lookup conveniences, not persistent + key identities. See [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). +- [Pending dispatch](../../packages/stack-encrypt/src/target/pending.rs) + currently takes one selected keyset for generates and groups retrieves by + their stored keyset IDs. It constructs generate/retrieve payloads without + lock context and passes no unverified request context. +- [Key-source traits](../../packages/stack-kms/src/key_source.rs) and + [payloads](../../packages/stack-kms/src/payload.rs) already model ZeroKMS + lock-context lists, optional decryption policies and unverified context. + They are ZeroKMS-specific abstractions used for provider injection/testing, + not yet a neutral contract for arbitrary KMS backends. Returned key types + also reflect the existing ZeroKMS/recipher integration. +- [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) + accepts `.withLockContext({ identityClaim: ["sub"] })`. Its documented + behavior is server resolution of claim values from the authenticating + principal. Encrypt/decrypt operations pass that selector to protect-ffi; + audit metadata travels separately as `unverifiedContext`. +- The pinned `zerokms-protocol` 0.12.31 `Context` enum includes `Tag`, + `Value` and `IdentityClaim`. Its `RetrievedKey` currently exposes key + material only; `stack-kms::DataKey` contains IV and key, not the original + authentication tag returned by the proposed refinement flow. Supporting + that flow remains a protocol/client response integration requirement, + consistent with the user's earlier explanation of intended behavior. + +## 1. Plaintext-derived keyset selection + +Add a key-selection declaration to the existing target definition, parallel +to context/scope source selectors. A useful minimal portable form is an +explicit ID or name selector over a plaintext field: + +```rust +// Proposed attribute fragment. tenant_id contains the actual keyset name. +#[stash(keyset(name(field = "tenant_id")))] +``` + +```rust +// Alternative: the selected field contains a keyset UUID, not an arbitrary +// application UUID assumed to be a keyset. +#[stash(keyset(id(field = "keyset_id")))] +``` + +Neither selector creates a keyset. Missing, unknown, unauthorized, disabled +or incorrectly typed keysets are errors; never fall back to the default. +A business tenant ID need not be a keyset name/UUID. Where they differ, +configure an explicit tenant-to-key mapping/resolver. Start with direct +selectors and a shared registered resolver contract; do not encode arbitrary +Rust callbacks in dynamic definitions. Pure mappings can return a selector; +network lookup belongs to the execution/provider layer, not the derive. + +The existing manually selected form already fits the agreed API: + +```rust +let tenant = cipher.keyset(IdentifiedBy::Name(user.tenant_id.clone().into())).await?; +let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; +``` + +The proposed derived route removes that repetition: + +```rust +// EncryptedUser's definition declares how to select the keyset. +let encrypted: EncryptedUser = user.encrypt_into(&cipher).await?; +let encrypted: Vec = users.encrypt_into(&cipher).await?; +``` + +`&cipher` is the client/provider executor which can resolve keysets; it is +not an unscoped request that silently encrypts under the default. Extend the +conversion executor capability to accept either a resolved keyset or a +client plus a declared selection recipe. The target's representation and +encryption operation description remain unchanged. Direct +`tenant_keyset.encrypt(value)` retains its accepted meaning. + +Execution order: + +1. Acquire guarded source ownership at request construction as already + required. Validate target/options, input shape, selectors, role/capability + requirements and every key-independent value check before provider I/O. +2. Evaluate the key-selection recipe from the guarded plaintext. Resolve + selectors through the configured provider; pin each resolved identity + for the duration of the request, including index derivation. +3. Partition a batch by resolved key identity, provider/authentication + session and compatible request-level options. Preserve original result + ordering; many tenants do not imply one remote call for the whole batch. +4. Use the selected keyset's index material for its searchable fields and + mint data keys in the selected keyset. Persist the resolved identity, + not a mutable name, in the ciphertext's provider metadata. + +Define conflict behavior: a preselected keyset cannot silently be overridden +by a target's route. If both are provided, validate that the resolved route +matches the bound keyset, or reject the combination explicitly. No implicit +precedence or fallback. Errors requiring name resolution can only occur +after that resolution; the accepted preflight rule covers key-independent +checks, not provider facts unavailable locally. + +For queries there is no plaintext user to inspect. The same target policy +must expose a caller-supplied tenant routing input or use an explicitly +selected keyset. A typed input may deliberately supply both key routing and +search scope; declaration determines both uses. Do not derive a keyset from +the email search value. Searching across multiple keysets is explicit +fan-out with a term per keyset, not one universally comparable query term. + +Decrypt uses the stable key identity stored with the ciphertext, as today. +Do not re-resolve a historical keyset name and substitute its current target. +An expected tenant/keyset restriction can constrain retrieval, with any +historical key mapping supplied explicitly. Merely trusting a tenant field +returned alongside a ciphertext does not authenticate application tenancy; +row binding or trusted expected context supplies that separate check. +Likewise, choosing a keyset from caller-controlled plaintext is routing, +not permission to use every tenant keyset accessible to a service account. + +## 2. Vitamin C AsyncCipher + +**Packaging follow-up:** the user favors removal. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md#retire-asynccipher) +now recommends retiring this trait and its application helpers as execution +moves into Stack, rather than adding the optional adapter considered below. +Vitamin C still has helper/test consumers to migrate; no removal has been +implemented. + +The absence of `AsyncCipher` is intentional in the current design, not an +unexplained omission. [cipher.rs](../../packages/stack-encrypt/src/cipher.rs) +documents the reason, and implements the synchronous Vitamin C `Cipher` +through `MintedKeys` (seal) and `BoundKeys` (open). + +Vitamin C 0.8.0 `AsyncCipher` takes leaf plaintext and derived AAD, with +`seal_many`/`open_many` operating on the leaves of one document. Stack's +executor retains the original provider descriptor and combines requests +across fields/documents/records before calling the leaf cipher. Leaf AAD +alone is not a reversible representation of those execution inputs. +Future routed keys and lock policies add more execution metadata rather +than removing the mismatch. The trait also requires `Sync` and `Send` +futures; Stack's provider traits use `MaybeSend` for native/WASM support. + +Recommendation: retain Stack's batch/provider orchestration and Vitamin C's +leaf cryptography. Do not implement `AsyncCipher` on `StackCipher` merely to +claim trait symmetry, encode provider policy secretly into AAD, or lose +cross-record batching. An explicitly configured adapter could implement the +trait for a restricted use case with provider descriptor/key route/policy +already supplied, but it is not required for the public redesign and must +advertise its limitations. A general refactor would need a richer request +contract and evidence that batching/metadata/WASM behavior remain intact. + +There is a smaller cleanup opportunity: each synchronous adapter currently +implements an unused half of `Cipher` by returning an error. Consider +separate seal/open capabilities or an internal adapter improvement during +the final dependency/packaging pass. Do not conflate that with requiring the +public Stack client to implement the async trait. + +## 3. Lock context as an optional provider policy + +Use a dedicated, typed ZeroKMS policy input, not the ordinary +`encrypt_with_context` argument and not a `SearchScope`. Proposed spelling: + +```rust +use stack_encrypt::providers::zerokms::{LockContext, LockContextExt}; + +let lock = LockContext::identity_claim("sub"); +let encrypted = user.encrypt_into(&tenant_keyset) + .with_lock_context(&lock) + .await?; +let user = User::decrypt_from(encrypted, &tenant_keyset) + .with_lock_context(&lock) + .await?; +``` + +This is illustrative module/method syntax pending the naming pass. The +operation builder retains owned plaintext guards while policy is attached. +The extension is present for a provider with the required capability; a +runtime-erased provider checks the equivalent capability before key I/O. +Ordinary encryption without this optional policy works on providers that +do not support it. A request/definition requiring it must fail on those +providers, never discard it or substitute literal AAD. + +The input selects a claim **name**. ZeroKMS verifies the authenticating +principal and resolves its claim value; do not decode an unverified JWT in +the application and turn its `sub` into a caller assertion. Authentication +belongs to the configured provider/session. Use explicit, immutable +request/session binding when identities differ concurrently; avoid mutating +a global client's identity between operations. + +Support explicit literal tag/value items through the same provider policy, +while distinguishing those caller assertions from principal-derived claims. +Missing claims or provider policy failures are errors with no retry without +the lock. Forward policy on generate and retrieve. A required policy can +live in the target definition so callers cannot accidentally omit it; an +operation may supply an undeclared optional policy, but cannot replace or +weaken a definition-required one. Define composition/conflict rules rather +than last-call-wins behavior. Policy selectors may need to be available on +reads; define their storage or required caller input without serializing +JWTs, credentials or claim values resolved by the provider. + +Keep audit metadata (`UnverifiedContext`) separate: it is not trusted +identity, ciphertext context or a policy assertion. Preserve distinctions +between lock context and the protocol's separate decryption-policy feature; +do not silently reinterpret one as the other. + +Index/query derivation may use an already-loaded deterministic index key +without generating/retrieving a data key. Therefore lock context on those +data-key endpoints does not by itself restrict who can derive search terms +or view query results. If search authorization is required, define that +capability separately; do not infer that the TypeScript query method's +acceptance of `withLockContext` proves the new index path is protected. +Do not implicitly make search scope principal-specific to emulate a lock. + +Batching must preserve per-item lock inputs and separate incompatible +principal/provider sessions. If a provider supports per-payload policy, +retain it in each payload; if an option is request-wide, partition by it. +Any future data-key cache must respect policy/principal checking semantics; +a cache hit must not silently replace a required provider authorization or +audit event. This is a design requirement, not a claim of an existing cache +vulnerability. Do not broaden the current keyset/index-key cache into an +unqualified data-key authorization cache. + +## Provider-neutral base and progressive capabilities + +The public value/target model needs extensions, not replacement. The larger +refactor is the provider boundary, which currently imports ZeroKMS payloads, +UUID keyset IDs, policies, errors and returned key types directly. + +Recommended capability split (working names): + +| Capability | Required for | +| --- | --- | +| Data-key generation/recovery and protected key material | Ordinary opaque and structural encryption/decryption | +| Index-key support | Searchable targets and query derivation | +| Keyset resolution by ID/name | ZeroKMS-style keyset selectors; not every provider | +| Lock-context support, including claim selectors | ZeroKMS policy extension; not every provider | + +Define a resolved provider key reference and versioned opaque recovery +metadata. Do not force every provider to supply a ZeroKMS UUID, IV/tag or +recipher response. A caller selects/configures a provider; ciphertext +metadata identifies a registered provider/key format, never arbitrary +network endpoints or credentials to execute. Keep ZeroKMS's key tag simple +to extract within its documented envelope profile for Rust/SQL consumers. +Other providers' metadata need not invent such a tag. Stable blob payload +encoding is independent of these provider-specific envelope profiles. + +Future AWS KMS/Vault integrations implement the minimal capabilities they +offer. They need not implement lock context, named keysets or searchable +index-key management merely to encrypt an opaque value. The application +can configure its own tenant-to-provider-key mapping without representing +it as a native ZeroKMS keyset. A searchable target requires an explicitly +supported index-key strategy; the engine must not assume every data-key +backend supplies deterministic search keys automatically. + +Refined-tag support also needs distinct stored/request tag and returned +original authentication-tag input. Preserve the user's intended ZeroKMS +flow when designing neutral responses; do not require that every provider +implements refinable tags or that a generic KMS returns that concept. + +## Dynamic interoperability + +Provider capabilities join target capabilities in discovery and definition +validation. Example provider-specific extension (proposed logical schema): + +```json +{ + "provider": "zerokms", + "key_selection": { + "keyset": { "by": "name", "encrypt": { "field": "tenant_id" } } + }, + "provider_policy": { + "zerokms": { + "lock_context": [{ "identity_claim": "sub" }] + } + } +} +``` + +The field's declared type must admit the chosen selector; missing fields +are definition/input errors before provider resolution. Query routing +declares an equivalent caller input because it has no source record. +Providers and extensions are registered, versioned capabilities. Unsupported +required key selection/policy is an error during loading or binding to an +execution provider, before key I/O. Do not accept arbitrary extension maps +and ignore unfamiliar entries. Auth credentials stay in provider setup, not +in this declaration or persisted policy metadata. + +Go generation uses named selectors/policies and the same validated dynamic +definition; provider-specific methods/capabilities are explicit. The +interoperability specification must include mixed-keyset batches, policy +inputs, capability refusals, key identity persistence and the distinctions +between authenticated claims and caller-supplied audit metadata. + +## Implementation and acceptance work + +1. Specify neutral key references/recovery metadata and optional capability + traits without altering the accepted encrypt/decrypt conversion roles. +2. Add definition selectors and client-scoped routed execution. Cover ID + versus name, explicit mappings, conflict errors and no default fallback. +3. Extend request planning/grouping to carry resolved route and provider + policy. Preserve per-record ordering and guard lifetimes across async + resolution/cancellation. +4. Wire ZeroKMS lock context through both generate/retrieve, including + literal and principal-claim forms, and test required-policy enforcement. +5. Verify mixed-tenant batches, matching write/query keys, historical ID + decrypt, wrong expected keysets, missing claims, unsupported-provider + failures before I/O, and concurrent principal isolation. A minimal fake + provider without keysets/locks/index keys must still encrypt opaque values. +6. Resolve refined-tag response support with the ZeroKMS client/protocol. + Add integration fixtures without assuming the pinned API already returns + the original tag. +7. Defer the `AsyncCipher` adapter/trait cleanup decision to the packaging + pass; preserve the documented batching and WASM constraints meanwhile. + +No implementation changes, protocol guarantees for future providers, or +completed security audit are claimed by this assessment. From c5efc48837d3cb48bc7e168f780fd616a4f4845e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 22:41:49 +1100 Subject: [PATCH 3/4] docs: prioritize Go contract and stable storage --- .../2026-10-24-stack-encrypt-api-plan.md | 101 ++++++++++-------- .../2026-10-24-stack-encrypt-api-review.md | 23 ++++ ...026-10-25-stack-encrypt-json-operations.md | 10 ++ .../2026-10-25-stack-encrypt-packaging.md | 36 ++++--- 4 files changed, 113 insertions(+), 57 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 600b52091..2109fe49f 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -10,6 +10,15 @@ criteria. User requirements are distinguished from recommendations that have not yet been accepted. Examples below are proposed APIs, not working code for the current release. +**Customer delivery priority, 2026-10-25:** the JSON-tree stack +headed by #1363 has merged; next deliver Go JSON and configurable Bloom search, including +the short-input requirement under investigation. Stabilize the Go contract +and persisted encoding/search semantics before customer release. Adopt the +Vitamin C codec in its existing repository now; defer crate moves and the +standalone Rust facade. The customer-first sequence below supersedes +earlier implementation ordering where they conflict. The short-gram design is +a release dependency, not a feature silently deferred to a later version. + **Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets are accepted. The comprehensive JSON first-release contract and Rust/dynamic examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), @@ -43,6 +52,9 @@ aliases, deprecation period, or data migration is required. Choose the coherent new design and update its Rust, Go, guest, and EQL consumers together. Engine output must still agree with the SQL that consumes it; that is current interoperability, not a legacy-format requirement. +This exemption ends for the customer profiles when released: subsequent +Rust API/package refactors must preserve the Go interface and stored +data/search compatibility. ## Requirements established in the review @@ -59,8 +71,10 @@ that is current interoperability, not a legacy-format requirement. prototype-compatibility reason to add a raw-byte exception. - Rust traits and derives are the primary static interface. Plans support those declarations and dynamic/FFI lowering. -- Design and deliver static and dynamic support together. Go support is an - acceptance criterion, not a later follow-up. +- Design static and dynamic support against the same engine/definition + contract. Deliver the customer Go interface first; polished public Rust + traits/derives may follow. Engine-level Rust interoperability tests remain + required, so this does not introduce a second Go encryption implementation. - Provide decrypt entry points with and without explicit context. The no-context variant must not invent a default required application value. - `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation @@ -881,38 +895,32 @@ this design discussion. ## Configured-target implementation sequence and completion criteria -1. **Normalize the general contract.** Specify typed options, Rust factory - signatures, validated dynamic deserialization, defaults, context - inputs, capability discovery, errors, and the target - object grammar. Create the versioned interoperability specification, - schemas and initial conformance fixtures described below alongside this - work. Build one target-owned validator and lowering path. -2. **Deliver JSON as a complete first implementation.** Add configured EQL - JSON targets, direct typed output/input conversion, catalog/guest - registration, and JSON transport. Keep engine and database encodings aligned. -3. **Expose both declarative front ends.** Add the generic Rust derive - factory hook and Go generator/policy support, including field targets and - standalone Rust wrappers. Neither front end reimplements encryption. -4. **Complete queries and reads in both languages.** Carry the configured - target into query construction, expose extraction results, and standardize - Rust's paired decrypt convenience methods. Settle the role-based JSON - names as part of this public API work. -5. **Verify the complete slice before calling it supported.** Rust-written - EQL JSON must open and query from Go, and vice versa, against PostgreSQL. - Test defaults, ordered paths, both root settings, arrays/scalar roots/null, - numeric boundaries, missing paths, context mismatch, and invalid options. - Assert logical batching and verify fixtures for the chosen format. Compare - semantics rather than ciphertext equality, since encryption is randomized. - Run the shared interoperability conformance suite and gate schema/runtime - drift in CI. Each implemented contract change updates its specification - and fixtures in the same change. - -These are implementation steps, not permission to ship a Rust-only phase. -The Go examples and cross-language acceptance tests are part of the feature. -Do not add the plaintext-configuration wrapper in this first change: the -dynamic target declaration already handles runtime options without adding -another policy location. Reconsider that convenience after both front ends -use the shared contract. +Customer commitments supersede the earlier Rust/Go front-end lockstep sequence: + +1. **JSON engine baseline is landed:** #1345, #1358, #1361 and #1363 have + merged. Build Go support against that baseline; revise the older #1353 plan. +2. **Settle Go/dynamic contracts and storage profiles:** target options, + context/scope, errors, query composition and stable Vitamin C encodings. + Vitamin C #520 and #526 implement the Rust/Go storage profiles and await + colleague review and release; adoption remains a Stack integration task. Start short-gram research immediately; + its storage/query implications must resolve before that feature freezes. +3. **Deliver Go JSON and configurable match search:** use the landed engine + and SQL contract, implement analyzer/filter definitions and supported + two-character behavior, and preserve the comprehensive JSON operation + inventory. Carry through the numeric round-trip fix from #1366 (#1364). +4. **Freeze and release the customer surface:** Go generation/runtime tests, + Rust-engine/Go cross-opening, PostgreSQL queries, schema/semantic + conformance, and stable storage/search vectors are release gates. + The final Rust derive facade is not. +5. **Complete Rust ergonomics and packaging:** implement public traits, + derives and examples and move crates behind the frozen Go and data + contracts. Internal plans may change; customer data must remain usable. + +Keep shared validation and one execution engine throughout. Final static +Rust examples become executable acceptance tests in step 5; they need not +delay step 4. Each delivered contract change updates its specification and +fixtures in the same change. Do not add a second policy location through +an ad hoc plaintext-configuration wrapper. ### Required deliverable: dynamic interoperability specification @@ -990,7 +998,12 @@ deliverable; recommendations do not silently settle the open decisions. | Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | | Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | -### A. Settle the public method and trait structure first +### A. Consolidate the public Rust method and trait structure + +This is the Rust-facade workstream. Under the latest customer ordering, do +only the engine/contract changes needed by Go before its release; defer +standalone Rust ergonomics and trait relocation. The sections below organize +the full review scope, not the customer critical path. Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. Use the accepted destination-directed decrypt pair. Write @@ -1462,6 +1475,8 @@ Completion: all public examples are executable tests, static/dynamic cross-opening and querying pass against the selected storage format, batching checks pass, and the guest builds remain supported. No new API is described as implemented until its Rust and Go acceptance criteria are met. +These are full redesign completion criteria. The earlier Go release uses +the facade-independent Go, engine and storage gates listed above. ## Recommended order for the remaining design decisions @@ -1492,15 +1507,14 @@ specification and implementation obligations: | --- | --- | | JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | | Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | -| Stable blob profile | Specify supported Serde shapes, lossless options/enums/maps, authenticated codec versions and compatibility fixtures under Vitamin C #519. This is the immediate priority; format-preserving cleanup/timing improvements are follow-up work, with limitations documented honestly. | +| Customer storage profiles | Stabilize the Vitamin C encodings and semantics used by Go/JSON/search before release, coordinating #519 upstream. Full Rust blob conveniences can follow; unsupported shapes must remain explicitly unsupported. Crate relocation is deferred. | +| Configurable match and short input | Whole-word semantics are confirmed. Define the analyzer/token-filter catalog and named indexes/Boolean composition; choose the short-word mechanism only after leakage/performance assessment. Select the storage/query scheme before release. | | Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | -Next review the JSON operation document and its static/dynamic examples, -then turn the required inventory into implementation phases and formal -interoperability fixtures. Stable storage-format work is tracked upstream. -Finish remaining query integration details without reopening ORE equality. -The user has now requested the naming/package pass while JSON review -continues; that proposal is ready for review in the packaging companion. +Next reconcile the operation document and #1353 with the merged JSON +stack. Settle the Go contract and adopt the upcoming Vitamin C storage +release. Use the customer-facing conformance gates above for release. Packaging remains a proposed destination after Go +delivery; finish query integration without reopening ORE equality. Specification/implementation work still includes exact context/scope grouping and encoding, dynamic grammar and version identifiers, error codes, binary @@ -1525,4 +1539,5 @@ reason to reopen the question of whether writes and queries share a policy. Passthrough implementation, transcoder adapters, collection composition, trait relocation, documentation updates, and acceptance tests remain work items. They are not reasons to characterize the agreed API direction as -undecided. No backward-compatibility phase is needed for the prototype. +undecided. No backward-compatibility phase is needed for the prototype; +the customer release establishes a new Go and persisted-data commitment. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index f60a088e5..f72bf39e4 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1121,3 +1121,26 @@ adapters also depend on the existing value model and need separation from their primitives. Moving the frame implementation refines the earlier codec ownership recommendation; coordinate stable storage work under Vitamin C #519. Names and package moves remain proposals, not implementation changes. + +## 22. Customer-driven sequencing: Go JSON and match search first + +The customer needs Go JSON support and robust Bloom-filter handling with +tokenizers/token filters, including two-character input. Standalone Rust +release quality is secondary. Claude's JSON-tree PR stack headed by #1363 has now merged. Revise the +older Go plan in #1353 against it and the agreed API direction. Vitamin C +#520 and #526 implement the storage profiles; colleague review and release +are pending. Stack still needs to adopt that release. + +Freeze the public Go contract and stored binary/search semantics before the +customer release. Internal plans may change behind that interface. Adopt and +stabilize the Vitamin C codec in its current repository now; defer physical +crate moves, AsyncCipher retirement and final Rust derives. The earlier +prototype compatibility exemption does not permit breaking customer data +after this release. + +The supplied `Short-ngrams.md` explores short whole-word filters and variant +queries, which must be distinguished from arbitrary two-character substring +search. The current tokenizer permits length 2, but the current one-match +output and one/range-query-term shapes need expansion for named filters and +Boolean alternatives. Resolve the scheme's leakage, SQL representation and +performance before release; do not silently defer the customer requirement. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 2ede5410a..7ff0d6fd3 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -13,6 +13,16 @@ they are not a post-release phase. Internal implementation can still be split into PRs. The feature is complete only when every required row below has Rust, dynamic/Go, and applicable PostgreSQL coverage. +**Customer sequencing update:** the PR stack headed by +[#1363](https://github.com/cipherstash/stack/pull/1363) has merged. Deliver Go +against that baseline. Its token lists, self tags, wildcard probes and result semantics +supersede the older gated-walk baseline used in parts of this document. +Reconcile those recipes before implementation and format freeze; the +operation inventory remains required. Engine-level Rust interoperability +coverage is a release gate; the final public Rust facade may follow +customer delivery. Retain regression coverage for the numeric round-trip +fix now merged in #1366 (closing #1364). + This is a target API contract, not a claim that these public signatures already exist. Rust and JSON examples below propose the redesigned surface. Final names/package paths are deliberately left to the last naming pass. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 7b4e749d3..9c7d50e7c 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -7,6 +7,12 @@ renames. Existing companion examples retain their working names until this proposal is accepted. The detailed JSON operation contract remains under separate user review. +**Sequencing update:** customer Go delivery supersedes the original +move-first sequence. The JSON stack headed by #1363 has merged. Vitamin C +#520 and #526 address storage and await review/release. Adopt those encodings +and deliver Go JSON/match search first. The package map below is the intended destination; neither a new +format crate nor public Rust facade completion is a customer release gate. + Source baseline: the current documentation branch and inspected local Vitamin C 0.8.0 sources. Stack's [cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) @@ -229,20 +235,22 @@ or dynamic discriminants. ## Implementation order and acceptance -1. Approve this glossary and ownership map; update all public example imports - and type names together without implying the examples already compile. -2. Specify the stable format profiles and conformance fixtures, coordinating - #519. Establish the small format crate and move the encoding wrappers and - frame foundation; preserve distinct transport/storage semantics. -3. Move the value model, traversal and dependent normalization/index adapters - into Stack, separating primitive inputs so the dependency graph stays - acyclic. Move relevant tests with their implementations. -4. Introduce neutral provider capabilities and the ZeroKMS adapter, rename - the protocol crate, then retire Vitamin C's async application execution - surface. Keep mixed-keyset/auth-policy batching and request isolation. -5. Apply the agreed method/derive/JSON names across Rust, EQL, dynamic catalog, - guest ABI adapters, Go generation and examples. Preserve the comprehensive - first-release JSON inventory rather than treating packaging as a scope cut. +1. Use the merged JSON stack and specify Go-visible names/contracts and stable + profiles needed by the customer. Keep codec work in Vitamin C, coordinating + #519. Preserve distinct transport/storage semantics and freeze search + semantics as well as byte layouts. +2. Deliver and validate Go JSON, configurable match search and the selected + short-input solution under the customer-first sequence. Record frozen Go/data + fixtures; leave internal Rust plan ergonomics free to change. +3. After delivery, establish the shared format crate and move the value model, + traversal and dependent normalization/index adapters into Stack. Preserve + frozen encodings and an acyclic graph; move tests with their code. +4. Complete neutral provider extraction and protocol-package renaming, then + retire Vitamin C's async application execution surface. Required Go-facing + policy and routing contracts must already have been handled before release. +5. Complete public Rust trait/derive names, imports and executable examples. + These changes must not rename the shipped Go API, reinterpret stored + profiles or require customer re-encryption/reindexing. Acceptance includes an acyclic Cargo graph; an EQL wire-only build without the engine; engine builds without ZeroKMS and without HTTP; native/guest From cddddda12c2a3d4f2dc2fbeab68cf5e28365e8d1 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 27 Oct 2026 13:03:54 +1100 Subject: [PATCH 4/4] docs: reconcile API plan with merged Go work --- .../2026-10-24-stack-encrypt-api-plan.md | 260 ++++--- .../2026-10-24-stack-encrypt-api-review.md | 30 + ...026-10-25-stack-encrypt-json-operations.md | 715 +++++------------- .../2026-10-25-stack-encrypt-packaging.md | 37 +- ...-25-stack-encrypt-provider-capabilities.md | 9 + 5 files changed, 395 insertions(+), 656 deletions(-) diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md index 2109fe49f..00969faf7 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-plan.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -10,14 +10,28 @@ criteria. User requirements are distinguished from recommendations that have not yet been accepted. Examples below are proposed APIs, not working code for the current release. -**Customer delivery priority, 2026-10-25:** the JSON-tree stack -headed by #1363 has merged; next deliver Go JSON and configurable Bloom search, including -the short-input requirement under investigation. Stabilize the Go contract -and persisted encoding/search semantics before customer release. Adopt the -Vitamin C codec in its existing repository now; defer crate moves and the -standalone Rust facade. The customer-first sequence below supersedes -earlier implementation ordering where they conflict. The short-gram design is -a release dependency, not a feature silently deferred to a later version. +**Implementation reconciliation, 2026-10-27:** reviewed merged `next` at +`ffeed7adb3f7d8739cda53261046147852bffc82`. Go JSON catalog, configured +targets, queries, request-bound reads, typed documents and executable docs +have landed (#1372, #1373, #1378, #1402). Opaque Go storage adoption has +landed (#1369/#1370). The active customer work is now contract conformance, +remaining read/configuration gaps and release readiness, not implementing +those foundations again. See the refreshed +[Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and [JSON inventory](2026-10-25-stack-encrypt-json-operations.md). + +#1379 added cascade authentication and randomized storage/projection order. +Wildcard `->` now selects an unspecified element; old first-in-document-order +recipes are invalid. Freeze the post-cascade format and compare query +semantics rather than randomized byte ordering. #1397 implements short +whole-word terms in the shared Bloom filter; configurable native Go match +options and broader tokenizer/filter work remain. This is implementation +status, not acceptance of a stronger privacy claim for those terms. + +Go's Vitamin C storage modules are temporarily pinned to `06568ba93bd9`. +The final-release guard requires replacing pseudo-versions with released +module tags; prereleases may carry them. Rust remains on crates.io 0.8.0. +Keep crate moves and the final Rust facade behind customer delivery. **Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets are accepted. The comprehensive JSON first-release contract and Rust/dynamic @@ -52,7 +66,9 @@ aliases, deprecation period, or data migration is required. Choose the coherent new design and update its Rust, Go, guest, and EQL consumers together. Engine output must still agree with the SQL that consumes it; that is current interoperability, not a legacy-format requirement. -This exemption ends for the customer profiles when released: subsequent +The merged Go implementation nevertheless retains a legacy JSON opaque +reader; record it as existing behavior, not a new requirement to add +prototype migration machinery. This exemption ends for customer profiles when released: subsequent Rust API/package refactors must preserve the Go interface and stored data/search compatibility. @@ -202,15 +218,15 @@ particular, the full ordered-path policy is not stored in the JSON header. ## Candidate dynamic representation and Go requirements -Extend the existing string target with an object form, using the current -type name provisionally rather than deciding a registry rename here: +Implemented by #1372: string targets and configured object targets share +the native options validator. Use the landed `JsonSearch` identifier: ```json { "payroll": { "context": ["employees", "payroll"], "target": { - "name": "JsonTreeDocument", + "name": "JsonSearch", "options": { "ordered": ["$.salary"] } } } @@ -218,7 +234,7 @@ type name provisionally rather than deciding a registry rename here: ``` Keep `"target": "TextEq"` as shorthand for that target's default options. -The object form is new: current dynamic targets accept only a string. +The object form and JSON target options are now implemented. Reject unknown options, incompatible options, unsupported target names, and invalid paths during declaration validation. Do not silently fall back to defaults. Specify one consistent JSON wire format and context encoding @@ -236,13 +252,13 @@ serialize its resolved options, not its function identifier. This contract does not require all Rust-only targets to depend on Serde; data conversion is required for targets registered with the dynamic interface. -Current gaps: JSON tree is absent from EQL's catalog/target table, the -dynamic JSON index is refused with `JsonIndexPending`, target fields carry -only a name, and Go's generated target declarations carry no target options. -Supporting an index's option syntax in Go does not establish executable -JSON-tree support. Include JSON input transport for objects, arrays, scalar -roots, and null; preserve the engine's numeric semantics and reject values -the cross-language contract cannot faithfully represent. +The catalog/guest, JSON value kind, strict JSON text parser and generated +Go target options are now implemented. #1373 adds operation input shapes and +operand encodings, and #1378 adds typed Go document adapters. Remaining +work includes the formal versioned specification, generalized definition +lifecycle/caching, direct typed EQL conversion and context-role separation. +The current data-plan grammar is still a field map; the illustrative +versioned schema below is not already accepted by the runtime. Writes and queries must reuse the normalized configured target in Rust and Go. An ordering query on an undeclared path must fail before key I/O. @@ -408,48 +424,34 @@ and batching; this trait separation must not add a blanket `Clone` bound. The final signatures must show where borrowed and owned sources are supported rather than concealing the existing difference behind the rename. -## Parallel Go JSON work: PR #1353 - -Read on 2026-10-24: [PR #1353](https://github.com/cipherstash/stack/pull/1353) -is an open documentation-only plan, not an implementation. Its write-up is -[The json_tree capability in the Go SDK](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md). -It tracks [#1347](https://github.com/cipherstash/stack/issues/1347), with -catalog, dynamic execution, queries, Go SDK, and extraction work in -[#1348](https://github.com/cipherstash/stack/issues/1348), -[#1349](https://github.com/cipherstash/stack/issues/1349), -[#1350](https://github.com/cipherstash/stack/issues/1350), -[#1351](https://github.com/cipherstash/stack/issues/1351), and -[#1352](https://github.com/cipherstash/stack/issues/1352). - -Treat that work as an implementation dependency to reconcile, not a parallel -SDK design to recreate. Its engine-owned execution, catalog/generated Go -types, engine-validated options, and cross-language tests fit this plan. -Review the following differences before freezing interfaces: - -- Route its JSON-specific option plumbing through the general target's - typed options/deserialization boundary. Go tags may retain idiomatic - syntax, but lower to target options rather than a second JSON policy path. -- Adapt the proposed `Layout` integration to the configured-target contract; - no second executor or permanently competing public configuration model. -- Reconcile type/registry names and generated query accessors with the shared - glossary and configured query capabilities. Its `At.*` accessor proposal - is useful input, not an accepted Rust or Go naming decision here. -- Carry the agreed context requirements and paired method semantics into - the relevant language interfaces without requiring identical language - syntax. Distinguish JSON `null` as content from nullable SQL fields. -- Evaluate its JSON-text ABI recommendation for numeric fidelity and return - types. This is compatible with direct typed EQL output adaptation: input - transport text and an avoidable encrypted-output Serde round trip are - different boundaries. -- Release scope is now decided: extraction, opening entries and walks ship - in the comprehensive first version alongside writes/queries. The earlier - Go plan's step-5 deferral is superseded by the linked JSON operation - inventory; sequential internal PRs must not become a reduced release. -- Byte-for-byte parity tests need deterministic randomness as that plan - proposes; normal randomized encryption needs cross-opening/query tests. - -These are dependencies and recommendations recorded locally. No changes or -comments have been posted to the parallel PR or its issues. +## Go JSON implementation and PR #1353 + +The [Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +has been updated against merged `next`. Tracking issue #1347 and steps +#1348–#1352 are closed. #1372, #1373 and #1378 implement the catalog, +configured JSON targets, dynamic execution, operation-specific operands, +request-bound opening and generated typed Go frontend; #1402 supplies +current docs and the executable database example. + +The shipped target identifier is `JsonSearch`; Go uses `eql.JSONSearch`, +`JSONSearchQuery`, `JSONSearchLeafQuery` and `JSONEntry`. Do not impose the +proposed `EncryptedJson` glossary as a pre-release rewrite. Preserve the +role distinctions and freeze actual customer names before later Rust renames. +Prepared Go reads retain their cipher and use `Open(ctx, rows)`. The engine +rederives trusted selectors/read keys from the original request. No new +persistent guest-handle protocol is needed to implement that verification. + +Residual requirements must be explicit: SQL children/counts exist but a +parent-bound plaintext child opener does not; selector opening accepts at +most one entry per document group. Use subtree reads for all-match plaintext +retrieval. General scope/row-binding inputs remain unsupported for EQL +targets. The existing column/keyset profile can ship without claiming them. + +The JSON contract now uses randomized projection order and cascade-bound +ciphertexts. Semantic parity and cross-opening are required; raw term byte +equality is meaningful only with controlled test randomness. Go typed JSON +adapters follow `encoding/json`, while raw JSON goes through the strict +engine parser. Do not claim arbitrary Go encoders preserve every input byte. ## Prior query design: baseline and reconciliation @@ -834,44 +836,34 @@ Track this action in for this review. It explicitly distinguishes storage stability from the transport-only rationale in Vitamin C #332. -**Recommendation, 2026-10-25:** reuse Vitamin C's frame writer/parser as the -foundation for the library-owned blob codec, with a specified blob Serde -mapping. Do not adopt the existing transport Serde mapping unchanged. -The inspected dependency is `vitaminc-aead-frame` 0.8.0; its `serde` module -already provides `to_vec(&T)` and `from_slice::(&[u8])`. Stack currently -uses that module for dynamic control data. Its module documentation -explicitly scopes it to transport data rather than plaintext to seal. - -The implementation already guards the serializer output, temporary variant -buffers, and buffered map keys with `Zeroizing`. The underlying -`vitaminc-aead` frame writer grows into a fresh allocation and wipes the old -one. However, `to_vec` returns an ordinary `Vec` on success. Provide an -internal guarded-output entry point or sensitive sink for the blob path, -keeping the result guarded through sealing. Inspect remaining temporary -storage and custom Serde paths before asserting a complete cleanup contract. - -The current mapping is not lossless for every ordinary Serde type: -`serialize_some` delegates directly to its value, while `None` and unit both -encode as null. Thus `Some(())` reads back as `None`; nested optional values -have the same ambiguity. The blob mapping must preserve option presence -explicitly, using an unambiguous representation, and test these cases. -The current map representation also requires text keys. Specify support or -explicit rejection for non-string map keys, and document enum/buffered Serde -limitations rather than advertising all `Serialize` types as round-trippable. - -Frame parsing validates one complete frame before invoking `Deserialize`, -but that neither prevents type-level decode failures nor cleans up partial -application values. Resolve that cleanup boundary alongside format limits, -authenticated codec versioning, and cross-language fixtures. Reusing the -frame format is not a constant-time guarantee and does not make an opaque -blob a structurally encrypted JSON document. - -[Borsh](https://docs.rs/borsh/latest/borsh/) remains a fallback if the blob -mapping would require disproportionate changes. It introduces -`BorshSerialize` / `BorshDeserialize` instead of the Serde traits, and has -writer-based serialization. Changing serialization libraries does not by -itself establish wiping or a constant-time contract. Prefer evaluating the -existing frame machinery against these requirements first. +**Implemented upstream and in Go, 2026-10-27:** Vitamin C's dedicated +storage profile and Go reader/writer address the format work from #519 +(#520/#526, carried onto upstream main via #544). Stack #1369/#1370 adopt +the Go codec: opaque generated structs call `vcffi.MarshalStorage` and +`UnmarshalStorage`. The profile is versioned (`VCS` plus version), supports +explicit optional values and typed map keys, and is separate from transport. +The transport mapping's `Some(())` ambiguity and text-only map-key rules +must not be reported as defects of this new storage profile. + +Go generated shapes use `vc` tags and `GeneratedVersion2`. A reader rejects +unsupported future blob versions distinctly and retains the old JSON +fallback. Go storage mapping has documented qualifications: nil containers +become empty, supported nonfinite floats round-trip, and unsupported opaque +types are refused by generation. Go pointers and Rust `Option` do not always +write identical bytes; cross-reading and the normative corpus matter. + +Rust workspaces still use crates.io Vitamin C 0.8.0; the final proposed Rust +`OpaqueBlob` facade has not thereby been implemented. Adopt the dedicated +storage entry points when that facade lands rather than repurposing the +transport Serde codec. Go's temporary pseudo-version dependency must move +to released module tags for a final Go release. A new serializer evaluation +is not on the customer critical path. + +Stable bytes do not establish constant-time behavior or complete cleanup. +Retain the remaining guarded-output, temporary-buffer, partial decode and +caller-owned-value work, without changing stored format. JSON tree retains +its own node/envelope encoding; its post-cascade compatibility profile must +be frozen separately from opaque blobs. The same consuming contract applies to target-directed `encrypt_into`. For a structural target, declared passthrough fields intentionally @@ -895,32 +887,34 @@ this design discussion. ## Configured-target implementation sequence and completion criteria -Customer commitments supersede the earlier Rust/Go front-end lockstep sequence: - -1. **JSON engine baseline is landed:** #1345, #1358, #1361 and #1363 have - merged. Build Go support against that baseline; revise the older #1353 plan. -2. **Settle Go/dynamic contracts and storage profiles:** target options, - context/scope, errors, query composition and stable Vitamin C encodings. - Vitamin C #520 and #526 implement the Rust/Go storage profiles and await - colleague review and release; adoption remains a Stack integration task. Start short-gram research immediately; - its storage/query implications must resolve before that feature freezes. -3. **Deliver Go JSON and configurable match search:** use the landed engine - and SQL contract, implement analyzer/filter definitions and supported - two-character behavior, and preserve the comprehensive JSON operation - inventory. Carry through the numeric round-trip fix from #1366 (#1364). -4. **Freeze and release the customer surface:** Go generation/runtime tests, - Rust-engine/Go cross-opening, PostgreSQL queries, schema/semantic - conformance, and stable storage/search vectors are release gates. - The final Rust derive facade is not. -5. **Complete Rust ergonomics and packaging:** implement public traits, - derives and examples and move crates behind the frozen Go and data - contracts. Internal plans may change; customer data must remain usable. - -Keep shared validation and one execution engine throughout. Final static -Rust examples become executable acceptance tests in step 5; they need not -delay step 4. Each delivered contract change updates its specification and -fixtures in the same change. Do not add a second policy location through -an ad hoc plaintext-configuration wrapper. +Customer commitments supersede the earlier Rust/Go front-end lockstep sequence. +The foundational Go JSON slice and opaque storage adoption are now merged. + +1. **Specify the implemented contract.** Capture target options, operator + input/result encodings, grouped reads, errors and the column/keyset + profile in the formal interoperability spec and conformance fixtures. + Keep proposed schema envelopes distinct from the current field-map grammar. +2. **Close actual remaining customer gaps.** Parent-bound child opening, + complex ordered-path declarations if needed, configurable Go match + options (#1112) and the tokenizer/filter pipeline. #1397 already supplies + short whole-word matching through EQL; confirm its accepted leakage and + freeze its index semantics rather than planning it as unimplemented. +3. **Validate and freeze the release candidate.** Run the existing engine, + Go/guest and PostgreSQL suites against the selected bundle. Preserve + numeric/null behavior, cascade tamper checks, wildcard/fallback semantics, + known omission limitations and post-cascade stored vectors. Tests already + landed are not proof this review reran a release build. +4. **Release the dependencies and customer surface.** Replace Vitamin C's + temporary Go pseudo-version pins for a final release; the existing guard + allows prereleases. Opaque storage is already integrated; keep it separate + from the JSON-tree storage profile. Build production guests and matching SQL. +5. **Complete Rust ergonomics and packaging.** Implement public traits, + derives, definition lifecycle and crate moves behind the frozen Go/data + contract. Direct EQL conversion is a format-preserving optimization. + +Use one execution engine throughout. The final Rust facade and package +layout are not gates for the existing Go JSON profile. Do not reimplement +configured JSON options or request-bound reads that have already landed. ### Required deliverable: dynamic interoperability specification @@ -1505,16 +1499,16 @@ specification and implementation obligations: | Remaining question | Decision required | | --- | --- | -| JSON contract review | User review is in progress. Review concrete operation recipes against the comprehensive required inventory, including stricter path validation and request-bound extraction. Custom-stage portability, validation timing and initial ORE equality policy are now accepted. | +| JSON contract completion | Main Go surface is implemented. Close parent-bound child opening and formal conformance gaps; use current cascade/random-order semantics. Do not reopen operator design or rename shipped Go APIs for cosmetic consistency. | | Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | -| Customer storage profiles | Stabilize the Vitamin C encodings and semantics used by Go/JSON/search before release, coordinating #519 upstream. Full Rust blob conveniences can follow; unsupported shapes must remain explicitly unsupported. Crate relocation is deferred. | -| Configurable match and short input | Whole-word semantics are confirmed. Define the analyzer/token-filter catalog and named indexes/Boolean composition; choose the short-word mechanism only after leakage/performance assessment. Select the storage/query scheme before release. | +| Customer storage profiles | Go opaque storage is integrated. Replace temporary Go dependency pins for final release and freeze post-cascade JSON/search profiles. Rust blob conveniences and crate relocation may follow. | +| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenizers/filters remain, including #1112. Reconcile the earlier privacy concern explicitly; merging support is not proof of unchanged leakage. | | Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | -Next reconcile the operation document and #1353 with the merged JSON -stack. Settle the Go contract and adopt the upcoming Vitamin C storage -release. Use the customer-facing conformance gates above for release. Packaging remains a proposed destination after Go -delivery; finish query integration without reopening ORE equality. +Next complete the supported-contract specification and remaining customer +configuration/read gaps, then validate the candidate and resolve final-release +pins. The original JSON vertical slice is already implemented. Packaging +remains deferred; keep initial ORE equality policy unchanged. Specification/implementation work still includes exact context/scope grouping and encoding, dynamic grammar and version identifiers, error codes, binary diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md index f72bf39e4..4fa7e186d 100644 --- a/docs/plans/2026-10-24-stack-encrypt-api-review.md +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -1144,3 +1144,33 @@ search. The current tokenizer permits length 2, but the current one-match output and one/range-query-term shapes need expansion for named filters and Boolean alternatives. Resolve the scheme's leakage, SQL representation and performance before release; do not silently defer the customer requirement. + +### Merged implementation reconciliation, 2026-10-27 + +Reviewed `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` against the +previous `ebdb5de0a` baseline. #1372/#1373/#1378/#1402 deliver the Go JSON +catalog, configured targets, operators, request-bound opening, typed document +adapters and executable documentation. #1347–#1352 are closed. #1369/#1370 +integrate Go opaque storage; final Go releases still require replacing the +temporary Vitamin C pseudo-version pins. Rust dependencies remain on 0.8.0. + +#1379 adds cascade authentication and randomized stored/projection order. +Wildcard `->` returns an unspecified match, not the first document element. +Post-cascade data is the format to freeze; unchanged header version 1 does +not imply compatibility with the old prototype AAD. Read verification still +does not authenticate result completeness, absence or database row identity. + +#1397 implements deterministic, domain-separated short-word terms inside +the existing Bloom filter, enabled by EQL's pinned match profile. This avoids +a separate short-word index but is not the random-variant mechanism explored +earlier, and it does not by itself resolve the user's frequency/leakage +concern. Native Go index options still refuse configuration (#1112). The +uncommitted ngram research and Go delivery document are unchanged. + +The refreshed API plan and JSON operation inventory mark landed work as +implemented. Remaining items include the formal interoperability spec and +conformance corpus, parent-bound child opening, configurable Go match +pipelines, direct typed EQL conversion and later context-role separation. +The customer release does not wait for final Rust traits or crate moves. +This review inspected source and existing tests; runtime/database suites +were not rerun and no production code was changed. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md index 7ff0d6fd3..dc6d2e7e0 100644 --- a/docs/plans/2026-10-25-stack-encrypt-json-operations.md +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -1,189 +1,186 @@ -# JSON tree: first-release operation contract and public examples - -Date: 2026-10-25. Companion to the -[API plan](2026-10-24-stack-encrypt-api-plan.md) and -[public workflow examples](2026-10-24-stack-encrypt-public-examples.md). - -## Status and scope - -The customer requires a comprehensive first version. This document defines -the required operation inventory for Rust and the dynamic interface used by -Go. Extraction, opening extracted values, and subtree walks are included; -they are not a post-release phase. Internal implementation can still be -split into PRs. The feature is complete only when every required row below -has Rust, dynamic/Go, and applicable PostgreSQL coverage. - -**Customer sequencing update:** the PR stack headed by -[#1363](https://github.com/cipherstash/stack/pull/1363) has merged. Deliver Go -against that baseline. Its token lists, self tags, wildcard probes and result semantics -supersede the older gated-walk baseline used in parts of this document. -Reconcile those recipes before implementation and format freeze; the -operation inventory remains required. Engine-level Rust interoperability -coverage is a release gate; the final public Rust facade may follow -customer delivery. Retain regression coverage for the numeric round-trip -fix now merged in #1366 (closing #1364). - -This is a target API contract, not a claim that these public signatures -already exist. Rust and JSON examples below propose the redesigned surface. -Final names/package paths are deliberately left to the last naming pass. -The [packaging glossary](2026-10-25-stack-encrypt-packaging.md#json-names-by-role) -now proposes that pass; this document retains its working names until review. -They are candidates for executable documentation during implementation; -they have not been compiled against today's API. Existing SQL spellings and -current behavior are separately identified. Do not publish these as current -SDK instructions until the acceptance suite runs them. - -Sources reviewed: - -- [Claude's Go JSON plan, PR #1353](https://github.com/cipherstash/stack/pull/1353), - head `65ece530934d367e30ea99d1f149f76117191739`, still a draft when read. - Tracking #1347; steps #1348–#1352. Its step-5 deferral is superseded here. -- [JSON tree design](2026-10-17-json-tree-index.md), including the Phase 4 - notes on extracted-path checking and stored order. -- [Engine query implementation](../../packages/stack-encrypt/src/json/query.rs), - [path grammar](../../packages/stack-encrypt/src/json/path.rs), - [derivation/opening](../../packages/stack-encrypt/src/json/derive.rs). -- [EQL target and result types](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs), - [SQL functions](../../packages/eql/src/v3/json_tree/functions.sql), - [operators](../../packages/eql/src/v3/json_tree/operators.sql), - [refused operations](../../packages/eql/src/v3/json_tree/blockers.sql), - and [leakage contract](../../packages/eql/docs/reference/json-tree-leakage.md). - -The local source baseline is the reviewed `next` commit `b1049824b` and -this documentation branch. Untracked experimental EQL files are not used -as evidence of implemented JSON-tree behavior. - -## Roles and shared policy - -| Working name | Role | +# JSON tree: operation inventory and release contract + +Updated: 2026-10-27 against merged `next` +`ffeed7adb3f7d8739cda53261046147852bffc82`. Companion to the +[API plan](2026-10-24-stack-encrypt-api-plan.md). + +The Go JSON feature is implemented through #1372/#1373/#1378/#1402. This +inventory distinguishes current support from remaining requirements. The +public Rust derive/facade example below remains proposed, not executable +code for the current engine. The +[refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and its engine/Go findings document the exact source evidence. + +## Current names and policy + +| Role | Current public Go / dynamic / SQL name | | --- | --- | -| `JsonPlaintext` | Owned JSON input/output with `Zeroize`; an adapter around the engine's JSON value model, not ciphertext or a policy | -| `JsonOptions` | Validated target options: `sealed_root` and `ordered` paths | -| `EncryptedProfile` | Application-defined fixed-policy output wrapper | -| `JsonTreeDocument` | Existing EQL encrypted document envelope, stored in one column | -| `JsonTreeQuery` | Containment/path-equality query operand; cannot decrypt | -| `JsonTreeLeafQuery` | Ordering query operand; cannot decrypt | -| `JsonTreeSelector` | Encoded path selector, not a plaintext path or ciphertext | -| `JsonTreeEntry` | Extracted encrypted node with the metadata needed to open it | -| `JsonSubtreeKey` | Client-held walk request state: token, gates, requested path and policy binding | -| `JsonTreeSubtree` | Encrypted walk rows paired with their original request state | -| `JsonSelection` | Proposed owned plaintext result distinguishing missing, one value and multiple matches; implements `Zeroize` | - -`JsonPlaintext` is a proposed library adapter, not a claim that -`serde_json::Value` implements `Zeroize`. It accepts JSON text without -requiring callers to implement their own cleanup trait. Returned plaintext -is owned by the caller. Future zeroization/timing work must not change the -stored representation. - -One immutable definition supplies options, column identity, search scope -and ciphertext binding to every operation. Rust's derive builds it from a -typed options factory; dynamic loading deserializes the same options and -validates the same constraints. Query code never supplies a second options -object that can disagree with the write policy. - -For a row-bound, tenant-scoped field: - -- Ciphertext binding: declared column + tenant scope + row ID. -- Search selectors, value terms, ordering terms and walk-token derivation: - declared column + tenant scope, with the operation's domain separation. -- Queries do not require row ID. Decrypting a selected document, entry or - subtree requires it; SELECT examples must return that ID. -- Preserve JSON node selector/ordinal authentication in addition to these - outer bindings. The engine currently combines context roles; implementing - their separation is required, not just changing a derive attribute. - -## Definitive first-release inventory - -`q` below means the typed query result, not manually assembled EQL JSON. -SQL functions shown with `eql_v4` use the existing EQL v4 names. Parameter -types are explicit to avoid PostgreSQL selecting native jsonb operators. - -| ID | Required operation | Rust surface / result | Dynamic operation / result | Database use | -| --- | --- | --- | --- | --- | -| J01 | Encrypt document: object, array, scalar or JSON null | `value.encrypt_into` → encrypted target | `encrypt` → typed EQL document | Store `eql_v4_json_tree` | -| J02 | Decrypt complete document | `JsonPlaintext::decrypt_from` → JSON | `decrypt` → JSON text | Read full column | -| J03 | Batch write/read | `Vec<_>` conversion composition | `encrypt_many`, `decrypt_many` | Preserve row/result ordering; shared executor | -| J04 | Contains | `query(op::Contains, value)` → `JsonTreeQuery` | `contains` → same EQL operand | `profile @> $1::eql_v4.query_json_tree` | -| J05 | Contained by | `query(op::ContainedBy, value)` → `JsonTreeQuery` | `contained_by` | `profile <@ $1::eql_v4.query_json_tree` | -| J06 | Scalar equality at path | `query(op::AtPath(op::Eq), (path, scalar))` → `JsonTreeQuery` | `eq_at` | Same containment predicate as J04 | -| J07 | Four ordering predicates | `AtPath(Lt/Lte/Gt/Gte)` → `JsonTreeLeafQuery` | `lt_at`, `lte_at`, `gt_at`, `gte_at` | `(profile -> $1::text) < $2::eql_v4.query_json_tree_leaf`, etc. | -| J08 | Inclusive range | `AtPath(Between)` → two leaf operands | `between_at` → lower/upper operands | `>= lower AND <= upper` | -| J09 | Sort ascending/descending | `Selector` query → selector; no bound value | `selector` | `ORDER BY eql_v4.ord_term(profile -> $1::text) ASC/DESC` | -| J10 | Path exists | `Selector` query | `selector` | `json_tree_path_exists(profile::jsonb, $1::text)` | -| J11 | Extract first match | `Selector` query → optional encrypted entry from DB | `selector`; result accepted by `decrypt_entry` | `profile -> $1::text` or `json_tree_path_query_first` | -| J12 | Extract first match as text | Same selector; parse text into `JsonTreeEntry` | Same selector; parse encrypted entry text | `profile ->> $1::text`; never plaintext | -| J13 | Extract all matches | Same selector → encrypted entry rows | Same selector; `decrypt_entries` | `json_tree_path_query(profile::jsonb, $1::text)` | -| J14 | Open extracted scalar(s) | `decrypt_from` on source bound to selection request | `decrypt_entry`, `decrypt_entries` with selection handle | No server decrypt; container markers refused | -| J15 | Prepare subtree/positional request | `query(op::Subtree, path)` → `JsonSubtreeKey` | `subtree_key` → handle + SQL parameters | Token/gates, derived under field scope/options | -| J16 | Fetch/open subtree | Request-bound `JsonTreeSubtree`, then `decrypt_from` | `decrypt_subtree` with walk handle | `json_tree_subtree` / `json_tree_subtree_at` | -| J17 | Enumerate immediate children | J15 key, encrypted child rows | J15 key; opening scalar children uses J14 with walk validation | `json_tree_children(profile, token, gates)` | -| J18 | Count immediate children | J15 key; result is integer | J15 key; database returns integer | `json_tree_child_count(profile, token, gates)` | -| J19 | Walk budget and sealed-root fallback | Same J16 plaintext result regardless of path taken | Same handle and result shape | Default budget 64; zero walks without fallback | -| J20 | Compare stored documents/ordered entries within one scope | Existing typed EQL operands; no new plaintext term | Existing typed EQL operands | Document `@>`/`<@`; entry `< <= > >=`; same policy/scope required | - -J09–J13 and J17–J18 are SQL actions using a derived selector/key, not new -cryptographic primitives or SDK database execution methods. The dynamic -engine derives operands and opens results; the caller runs SQL. The -definition's capability discovery must describe both operand preparation -and the supported database use, including ordering-path restrictions. - -J20 does not authorize whole-document `=` or equality via an ordering -term. Initial ORE support never answers `Eq`. JSON scalar equality remains -available through its independent exact value-selector mechanism. Any -future ORE equality capability must be explicit and limited to encodings -that preserve equality; text/lossy encodings remain ineligible. - -## Path, null and result semantics - -- Grammar: `$`, `.member`, `["member"]`, nonnegative `[n]`, and `[*]`. - No filters, recursive descent or slices in this release. -- Selector-based operations address a path class across array elements. - Existing selectors erase positions (`[2]` has the same selector as - `[*]`). **Proposed first-class API rule:** reject positional paths for - `EqAt`, selector extraction/existence and selector ordering rather than - silently broadening them. Use the positional subtree route for exact - `[n]` retrieval. This is a deliberate stricter wrapper than today's engine. -- Wildcard equality means at least one matching element has that scalar - value. Containment preserves grouping within array elements; it is not - an ungrouped bag of independent terms. Multiple independent EqAt SQL - predicates do not imply the values belong to the same array element. - Use one containment needle to require that correlation. -- Equality accepts null, boolean, string and supported JSON number values; - objects/arrays are refused. Containers use containment, not an operator - labelled equality. At `$`, equality asks for that scalar root, not an - array containing it (which is containment semantics). -- Ordering is opt-in per path and accepts supported numbers/strings only. - No implicit ordering at undeclared paths. A wildcard ordering policy can - supply terms to many entries: `->` compares/sorts the first matching - entry, while `json_tree_path_query` can enumerate all and an explicit - SQL `EXISTS` chooses any-match semantics. Never present first-match as - any-match, or term equality as exact plaintext equality. -- A missing path gives no extraction row (or SQL NULL for first-match). - A present JSON null gives an encrypted entry that opens to JSON null. - SQL NULL for an absent document is a separate field-null policy, not an - encrypted JSON null. An empty object/array is neither missing nor null. -- JSON inputs cross the dynamic ABI as UTF-8 JSON text. No intermediate - conversion of JSON numbers to Go `float64`; use raw JSON/number-preserving - decoding. Define accepted numeric range and canonicalization from the - engine, test `2`/`2.0`, negative zero and large integers. Reject values - outside the supported representation rather than silently rounding them. -- SQL returns encrypted data, even with `->>`. A scalar node opens directly; - an ordinary container node is a marker, not a serialized subtree. Use a - subtree walk to reconstruct containers. The sealed root holds the whole - document and has an explicit whole-document/fallback decode path. -- Selection/open requests retain expected selector, field definition and - scope. Check the returned selector before opening: current code opens - under server-supplied selectors, which is insufficient to establish the - requested path. Row provenance requires explicit row binding. Walks must - validate the path/selector relationships needed for their requested result. - Successful decryption does not prove query completeness or authenticate - every search/index metadata field. -- Requests for `$` (or root-array wildcards requiring a root token) under - `sealed_root=true` use whole-document fetch/decrypt and local selection; - never fabricate a nonexistent root token. For a missing/empty/scalar - subtree where a descendants-only walk cannot distinguish the cases, use - the selected start-node metadata or full-document fallback explicitly; - do not treat an empty row set as an empty object by assumption. +| Configured target | `JsonSearch` | +| Encrypted document | `eql.JSONSearch`, `public.eql_v4_json_search` | +| Containment / EqAt operand | `eql.JSONSearchQuery`, `eql_v4.query_json` | +| Ordering operand | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | +| Encrypted entry | `eql.JSONEntry`, `public.eql_v4_json_entry` | +| Prepared selector / subtree | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | +| Singular plaintext result | `JSONValue{Found, JSON}` | + +Handwritten Rust EQL adapters retain their `JsonTree*` names; the internal +engine `JsonDocument` is not the Go ciphertext type. `eql.JSON` is the +storage-only type. Do not use the old `query_json_tree` / `eql_v4_json_tree` +domains in current recipes. + +A field's configured target options are resolved into native `JsonOptions` +and reused for writes, queries and reads. The shipped profile uses column +identity and keyset selection. Separate row binding and search scope remain +accepted future API work, not supported EQL context inputs. Reject those +unsupported declarations; do not append row IDs to the existing shared +context. Queries do not gain row isolation merely because their returned +ciphertexts authenticate. + +## Required inventory and current support + +| ID | Operation | Current behavior / remaining work | +| --- | --- | --- | +| J01 | Encrypt object/array/scalar/null | Implemented; raw JSON supports every root, typed adapters support declared structs/maps/slices | +| J02 | Decrypt complete document | Implemented; raw JSON text or declared Go type | +| J03 | Batch write/read | Implemented; shared executor and provider chunking | +| J04 | Contains | Implemented `Contains`, `contains`, `@>` with `query_json`; generated needle guards apply | +| J05 | Contained by | Implemented `ContainedBy`, `contained_by`, `<@` | +| J06 | Scalar equality at path | Implemented `EqAt`, `eq_at`; value selectors, not ordering equality | +| J07 | Four ordering predicates | Implemented runtime paths and `At.` accessors; ordered number/string paths only | +| J08 | Inclusive range | Implemented `Between`, `between_at`, named lower/upper operands | +| J09 | Sort | Selector plus explicit SQL; wildcard sort needs defined reduction, not an assumed first array element | +| J10 | Path exists | Selector plus `json_tree_path_exists` | +| J11 | Extract one match | Selector plus `->` / path-query-first; wildcard choice is unspecified | +| J12 | Extract as encrypted text | `->>`; still encrypted entry JSON, not plaintext | +| J13 | Enumerate all matches | SQL path query exists; bounded wildcard subtree is the supported grouped plaintext route | +| J14 | Open extracted scalars | `JSONSelection.Open` handles one optional entry per document group. Multi-entry selector groups are refused; do not label them supported | +| J15 | Prepare positional/wildcard read | Implemented `Subtree`, `subtree`; token list plus retained trusted request | +| J16 | Fetch/open subtree | `json_tree_subtree(document, tokens, budget)`, `JSONRead.Open` / `OpenOne` | +| J17 | Immediate children | `Children` prepares a concrete token; SQL returns child entries. Parent-bound child-scalar opening remains unimplemented | +| J18 | Count children | `ChildCount` token and SQL nullable count; not authenticated completeness | +| J19 | Budget/fallback | Implemented, with original requested selection applied after root fallback | +| J20 | Stored-value comparisons | Existing document containment and entry ordering, within compatible column/keyset/profile | + +The comprehensive release requirement remains. Do not equate closed tracking +issues with implementation of a parent-bound child opener, or feed child +rows into an unrelated selector/subtree request. Complete that opener for +the J17 plaintext-child requirement; until then, fetch and open the parent's +subtree to inspect its children, documenting the different transfer/leakage +cost. SQL all-match enumeration is distinct from the supported bounded +subtree plaintext interface. + +## Semantics that changed or are now implemented + +- #1379 randomizes stored entries, object ordinals and containment projection + order. `->` at a wildcard selects an unspecified element. Use all matches + with an explicit `EXISTS`, `MIN` or `MAX` rule for ordering/sorting. +- Cascade authentication commits containers to their child ciphertext tags. + Nonempty standalone container entries fail `JsonNotALeaf`, also used for + alteration; empty container entries authenticate but fail `OpenedType`. + All ordinary containers need a subtree read. A requested sealed root can + still open as a complete document. +- The dynamic engine now rejects positional selector/ordering/EqAt paths. + Concrete `[n]` positions belong to subtree reads; wildcard selectors address + a path class. Independent EqAt predicates need not match one array element; + one containment needle preserves their correlation. +- The strict Rust parser now rejects duplicate keys and out-of-range integer + literals before key I/O. Typed Go encoding occurs first and follows + `encoding/json`, including invalid-UTF8 replacement. Raw JSON is checked + directly. Neither path promises byte-for-byte JSON text preservation. +- SQL NULL, missing selection, JSON null and empty containers are distinct. + `JSONValue.Found` carries selection presence. `OpenOne` rejects wildcards; + plural subtree reads return requested values in document order. +- Prepared reads retain the cipher, field definition, path and wildcard + bound. `Open(ctx, rows)` asks Rust to rederive and verify the request. + No serialized guest-local key handle is required. Rows cannot supply their + own expected path. Keep one group per database document. +- Cascade strengthens verification beneath a returned root. It does not + authenticate the whole result set, absence, freshness or the intended + database row. Whole results/probes can still be withheld. +- Wildcard bound defaults to 64 with boundary probes and an 8,192-token cap. + Walk budget independently defaults to 64; zero disables root fallback. + An open-root document cannot use sealed-root fallback. The budget is not + a universal response-size cap. +- Header version remains 1 despite the changed cascade/AAD semantics. Freeze + the post-cascade profile explicitly; the unchanged header is not evidence + of compatibility with older prototype ciphertexts. Randomized query + projection order requires canonical semantic comparisons in parity tests. + +## Current SQL recipes + +The examples assume `profile public.eql_v4_json_search`. Parameters are the +SDK's generated query/selector/token values, never plaintext paths. Add the +application's authorization and tenant predicates as usual. + +```sql +-- Contains / contained by / EqAt operands. +SELECT id, profile FROM users WHERE profile @> $1::eql_v4.query_json; +SELECT id, profile FROM users WHERE profile <@ $1::eql_v4.query_json; + +-- A single-valued ordered path: selector, lower bound, upper bound. +SELECT id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Wildcard ordering: explicitly existential, not an unspecified -> match. +SELECT u.id FROM users AS u +WHERE EXISTS ( + SELECT 1 + FROM eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS x(entry) + WHERE x.entry > $2::eql_v4.query_json_leaf +); + +SELECT id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, profile -> $1::text AS entry FROM users; +SELECT id, profile ->> $1::text AS encrypted_entry_text FROM users; + +-- Token list, then walk budget. Group entries by id before opening. +SELECT u.id, w.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text[], $2::integer +) AS w(entry); + +-- Children/count take one concrete node token, with no gates parameter. +SELECT u.id, c.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children(u.profile, $1::text) AS c(entry); +SELECT id, eql_v4.json_tree_child_count(profile, $1::text) FROM users; +``` + +Child count is zero for scalars/empty containers and NULL for a missing node +or the sealed root. Do not convert NULL to zero. The former token/gates and +`json_tree_subtree_at` recipes are removed. Current full Go programs and +parameter binding examples live in +[the JSON example](../../languages/golang/encrypt/example/json/main.go). + +## Current dynamic grammar and results + +This is implemented data-plan syntax, without an illustrative versioned +outer envelope: + +```json +{ + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "JsonSearch", + "options": {"sealed_root": true, "ordered": ["$.age"]} + } + } +} +``` + +Operations select the field through the ABI. Their named inputs include +`{"json"}`, `{"path","json"}`, `{"path","low","high"}`, `{"path"}` +and `{"path","wildcard_bound"}`. JSON values are UTF-8 JSON text. Query +results identify `encoding` (`eql`, `text`, `text_array`, or scalar `bytes`), +SQL domain and operands. Capabilities identify input shapes. Opening uses +`json_open` / `se_json_open` with the original `selector` or `subtree` +request and grouped encrypted rows. The general versioned definition/schema +package remains required work; it is not the current runtime grammar. ## Rust write/read example @@ -255,326 +252,28 @@ requests. Store/return both ID fields for decrypt. If caller input is used, use the agreed `_with_context` variants with a typed input separating binding and scope. No row ID is ever an index/query input implicitly. -## Rust query and extraction recipes - -These snippets share `keyset` and a resolved definition `profile` from the -write example. Add `use stack_encrypt::{op, json::JsonPath};`. The `using` -selector chooses that definition before the operation. `Selector` and -`Subtree` are proposed operation markers integrating today's separate -`json_selector`/`json_subtree_key` helpers into the shared definition API. -All query operands below use the same plaintext JSON adapter as writes. - -```rust -let profile = EncryptedProfile::definition()?; -let contains = keyset.using(&profile) - .query(op::Contains, JsonPlaintext::parse(r#"{"name":"Alice"}"#)?) - .await?; -let contained_by = keyset.using(&profile) - .query(op::ContainedBy, JsonPlaintext::parse(r#"{"age":42,"name":"Alice"}"#)?) - .await?; -let eq = keyset.using(&profile) - .query(op::AtPath(op::Eq), ("$.name", JsonPlaintext::parse(r#""Alice""#)?)) - .await?; - -let lt = keyset.using(&profile) - .query(op::AtPath(op::Lt), ("$.age", JsonPlaintext::parse("65")?)).await?; -let lte = keyset.using(&profile) - .query(op::AtPath(op::Lte), ("$.age", JsonPlaintext::parse("65")?)).await?; -let gt = keyset.using(&profile) - .query(op::AtPath(op::Gt), ("$.age", JsonPlaintext::parse("18")?)).await?; -let gte = keyset.using(&profile) - .query(op::AtPath(op::Gte), ("$.age", JsonPlaintext::parse("18")?)).await?; -let range = keyset.using(&profile) - .query(op::AtPath(op::Between), ( - "$.age", (JsonPlaintext::parse("18")?, JsonPlaintext::parse("65")?), - )).await?; - -let age = keyset.using(&profile) - .query(op::Selector, JsonPath::parse("$.age")?).await?; -let visit_dates = keyset.using(&profile) - .query(op::Selector, JsonPath::parse("$.visits[*].date")?).await?; -``` - -The results carry named typed EQL payloads. For example, `lt.selector()` -and `lt` bind the selector and leaf operand in the ordering SQL below; -`range.lower()`/`upper()` name the two bounds (proposed accessors). -For static policies, generated ordered-path handles can add compile-time -checks; runtime paths still validate against the same definition before -term derivation. Missing definition/scope makes a Rust request unawaitable. - -```sql --- Containment/equality operands: bind the named query payload, not plaintext. -SELECT id, tenant_id, profile FROM users -WHERE profile @> $1::eql_v4.query_json_tree; - -SELECT id, tenant_id, profile FROM users -WHERE profile <@ $1::eql_v4.query_json_tree; - --- Lower and upper bounds; use <, <=, >, >= individually for J07. -SELECT id, tenant_id, profile FROM users -WHERE (profile -> $1::text) >= $2::eql_v4.query_json_tree_leaf - AND (profile -> $1::text) <= $3::eql_v4.query_json_tree_leaf -ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; - --- Existence, first encrypted entry, first entry as encrypted JSON text. -SELECT id, tenant_id FROM users -WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); -SELECT id, tenant_id, profile -> $1::text AS entry FROM users; -SELECT id, tenant_id, profile ->> $1::text AS entry_text FROM users; - --- All matching entries: group results by document before opening them. -SELECT u.id, u.tenant_id, e.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS e(entry); - --- Stored-to-stored comparisons require matching policy and search scope. -SELECT a.id, b.id FROM users AS a JOIN users AS b - ON a.tenant_id = b.tenant_id AND a.profile @> b.profile; -SELECT a.id, b.id FROM users AS a JOIN users AS b - ON a.tenant_id = b.tenant_id - AND (a.profile -> $1::text) < (b.profile -> $1::text); -``` - -Add the application's tenant filter where appropriate; cryptographic scope -does not replace SQL authorization. A selector identifies a path class; -the SQL above never receives `$.age` or another plaintext JSON path. - -Proposed request-bound opening preserves the expected path and avoids the -current server-supplied-selector ambiguity. Here `entry` and `entries` are -typed database results for the corresponding selection request, and each -batch belongs to the identified document/context: - -```rust -let value = JsonPlaintext::decrypt_from(age.bind(entry)?, &keyset).await?; -let values = Vec::::decrypt_from( - visit_dates.bind_many(entries)?, &keyset, -).await?; -``` - -`bind` is a proposed checked source adapter, not a second decrypt method. -For row-bound policies, call `decrypt_from_with_context` with that result -row's expected ID and required scope. Missing first-match rows stay absent; -do not manufacture an encrypted null. Containers use the walk route below. - -## Walks, children, positional retrieval and fallback - -```rust -let walk = keyset.using(&profile) - .query(op::Subtree, JsonPath::parse("$.visits[0]")?).await?; - -// After the SQL below returns all rows for one document: -let source = walk.bind_subtree(rows)?; -let visits = JsonSelection::decrypt_from(source, &keyset).await?; -``` - -`walk` retains the requested path, resolved scope and definition; only the -token/gates go to SQL. Exact positions are supported here, including the -necessary selector gate to distinguish an array element from an object -member at the same ordinal. The final specification must normalize result -cardinality through `JsonSelection`: a concrete path yields `Missing` or -`One(JsonPlaintext)`; wildcard paths yield `Many(Vec)` in -document order. Missing and an explicit JSON null remain distinguishable. -`bind_subtree`/opening must preserve that distinction. `JsonSelection` is a -working type name and is imported from the proposed JSON module. - -```sql --- $1: token; $2: gates; $3: budget (64 default, 0 disables fallback). -SELECT u.id, u.tenant_id, w.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_subtree_at( - u.profile, $1::text, $2::text[], $3::integer -) AS w(entry); - --- With no gates, the dedicated form is also supported. -SELECT u.id, u.tenant_id, w.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_subtree( - u.profile, $1::text, $2::integer -) AS w(entry); - --- Immediate children and counts use the same token/gates. -SELECT u.id, u.tenant_id, c.entry -FROM users AS u -CROSS JOIN LATERAL eql_v4.json_tree_children( - u.profile, $1::text, $2::text[] -) AS c(entry); -SELECT id, tenant_id, - eql_v4.json_tree_child_count(profile, $1::text, $2::text[]) -FROM users; -``` - -Use one request-bound source per document; do not merge entries from several -rows into a subtree. Immediate scalar children can use checked entry -opening; container children need their complete descendants or an explicit -full-document fallback. The SDK must expose this distinction rather than -pretend that a container marker decrypts to its children. - -Budget 64/0 are current SQL semantics to retain in the first public scope. -When `sealed_root=true`, an over-budget walk returns the sealed root instead -of partial data; the client opens it and selects the original requested -path. With `sealed_root=false`, walk the full result or report an explicit -resource limit, never silently truncate it. The same requested value(s) -must result from budget 0 and fallback. Tokens reveal subtree structure -within their search scope and are not decryption keys; document their -existing leakage and lifetime with these examples. - -## Dynamic definition and operation examples - -These are logical JSON messages for the specification, not a promise to -replace the frame ABI with JSON. The host/guest transport encodes their -typed equivalents. `using` identifies a validated field definition; the -engine, not Go, creates EQL envelopes and query terms. Naming the definition -in a request must resolve to the same options and scope contract as writes. - -The fixed-column example's dynamic counterpart is: - -```json -{ - "version": 1, - "name": "users.profile", - "plaintext": "json", - "target": { - "name": "JsonTreeDocument", - "column": { "table": "users", "column": "profile" }, - "options": { "sealed_root": true, "ordered": ["$.age", "$.visits[*].date"] } - }, - "search_scope": { "kind": "column" }, - "context": { "kind": "none" } -} -``` - -`version: 1` is an illustrative schema version, to allocate in the formal -specification. Named types are optional. For tenant/row policies use the -accepted `ScopeKey`/`PrimaryKey` declarations and source selectors; the -resolved query input requires the tenant scope but never the row ID. - -Request corpus (each array element is an independent example, not one -batch of mixed operations). JSON text is deliberately explicit so no host -float conversion precedes the engine parser: - -```json -[ - { "operation": "encrypt", "using": "users.profile", "value": { "json_text": "{\"age\":42}" } }, - { "operation": "encrypt_many", "using": "users.profile", "values": [{ "json_text": "{\"age\":42}" }, { "json_text": "{\"age\":27}" }] }, - { "operation": "query", "using": "users.profile", "operator": "contains", "input": { "json_text": "{\"age\":42}" } }, - { "operation": "query", "using": "users.profile", "operator": "contained_by", "input": { "json_text": "{\"age\":42,\"name\":\"Alice\"}" } }, - { "operation": "query", "using": "users.profile", "operator": "eq_at", "input": { "path": "$.name", "json_text": "\"Alice\"" } }, - { "operation": "query", "using": "users.profile", "operator": "lt_at", "input": { "path": "$.age", "json_text": "65" } }, - { "operation": "query", "using": "users.profile", "operator": "lte_at", "input": { "path": "$.age", "json_text": "65" } }, - { "operation": "query", "using": "users.profile", "operator": "gt_at", "input": { "path": "$.age", "json_text": "18" } }, - { "operation": "query", "using": "users.profile", "operator": "gte_at", "input": { "path": "$.age", "json_text": "18" } }, - { "operation": "query", "using": "users.profile", "operator": "between_at", "input": { "path": "$.age", "lower": { "json_text": "18" }, "upper": { "json_text": "65" } } }, - { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.age" } }, - { "operation": "query", "using": "users.profile", "operator": "selector", "input": { "path": "$.visits[*].date" } }, - { "operation": "query", "using": "users.profile", "operator": "subtree_key", "input": { "path": "$.visits[0]" } } -] -``` - -Responses carry a result kind and typed payload: document, containment -operand, leaf operand, range, selection, or walk request. SQL preparation -returns the named domain and encoded payload (or typed token/selector -parameters); it does not concatenate SQL. Selectors serve J09–J13, and walk -keys serve J16–J19, so those SQL uses do not need redundant engine opcodes. - -For decode operations, this table specifies the dynamic signatures. Values -such as `document` and `entries` mean complete validated EQL payloads, not -literal placeholder strings to parse as ciphertext: - -| Operation | Required inputs | Result | -| --- | --- | --- | -| `decrypt` | `using`, `document`, required context/scope | JSON text | -| `decrypt_many` | `using`, documents with per-record context/scope | JSON texts in input order | -| `decrypt_entry` | `using`, `selection_handle`, entry, required context/scope | Scalar JSON text | -| `decrypt_entries` | `using`, selection/walk handle, entries grouped by record and context | Ordered scalar JSON texts | -| `decrypt_subtree` | `using`, `walk_handle`, complete rows for one document, required context/scope | Explicit missing / one JSON value / multiple matches | - -The response to `selector`/`subtree_key` includes an opaque request handle -in addition to SQL parameters. The handle retains the path and resolved -definition/scope in the engine, so callers cannot accidentally substitute -arbitrary path/options during decode. Specify handle lifetime and release -in the guest ABI; handles are not persistent database values. If the host -must recreate one after restart, derive it again from the original request, -not from server-returned metadata. Stale or mismatched handles are errors. -An equivalent Rust source adapter retains the same state directly. - -Example host orchestration, using proposed dynamic method names rather -than inventing ciphertext strings in a JSON example: - -```rust -let loaded = dynamic.load_definition(definition_json)?; -let encrypted = loaded.encrypt(json_text, inputs).await?; -let stored = database.store(encrypted).await?; -let opened = loaded.decrypt(stored, decrypt_inputs).await?; - -let selection = loaded.selector("$.age", scope).await?; -let entry = database.select_first(selection.sql_parameters()).await?; -if let Some(entry) = entry { - let age = loaded.decrypt_entry(&selection, entry, row_inputs).await?; -} - -let walk = loaded.subtree_key("$.visits[0]", scope).await?; -let rows = database.subtree(walk.sql_parameters(), 64).await?; -let result = loaded.decrypt_subtree(&walk, rows, row_inputs).await?; -``` - -This is an orchestration recipe: `database` represents application SQL, -not a new Stack Encrypt database driver; the input variables are supplied -by that application. Generated Go methods should hide these message details, -use named results, and call the same validated dynamic operations. JSON -marshalling helpers must not force a containment needle into a whole record -struct whose zero-valued fields accidentally narrow the query. - -## Explicitly unsupported operations - -The comprehensive first release covers the inventory, not all PostgreSQL -JSON operators. Reject whole-document `=`/`<>` as plaintext equality, -entry equality through ordering terms, plaintext path SQL operands, -JSONPath filters/recursive descent/slices, and in-place encrypted document -patch/delete/concatenation. Updates read/decrypt, modify, then encrypt the -complete document. Projection extraction does not confer the ability to -write a standalone node back into an existing tree. - -Reject malformed JSON/path/options; unsupported numeric values; container -EqAt operands; boolean/null ordering operands; undeclared ordering paths; -selector-position ambiguity; missing scope/context; unknown/expired request -handles; and wrong definition/scope/path during opening. Preserve useful -reason codes and structural locations without echoing plaintext values. -Structural validation that needs no key must precede key I/O. The current -engine refusal names include `JsonValueNotScalar`, `JsonLeafNotOrdered`, -`JsonPathNotOrdered`, `JsonOrderingAtPosition`, `JsonTooDeep`, -`JsonTooWide`, `JsonSealedRoot`, and `OpenedType`; final dynamic codes must -map deliberately rather than exposing unstable Rust enum spellings. - -## Implementation gaps and release gates - -1. **Shared definitions and scope separation.** Implement fixed/dynamic - target resolution, typed options, scope-role validation, row-binding - inputs and matching write/query context derivation. Generate capability - metadata from one registry, including JSON-specific operand/result kinds. -2. **Catalog/guest coverage.** Complete the JSON tree catalog, target table, - guest registration, dynamic index execution and JSON-text record kind. - Current Rust helpers/EQL types are not evidence that Go supports them. -3. **Direct EQL adapters.** Replace the existing serialization round trip - with typed conversions for documents, entries and operands. -4. **Selectors and opening.** Add the request-bound source adapters and - dynamic handles, validate expected selectors, preserve required row - context, and test mismatched results and missing headers/ordinals. -5. **Walk completeness.** Cover children/counts, concrete positions, - wildcards, empty containers, missing paths, scalar starts, root requests, - sealed-root on/off and budgets 0/64. Resolve reconstruction ambiguities - explicitly, including returning/fetching the start node when needed. - Do not ship opening/walks later under the earlier PR #1353 scope. -6. **Conformance/documentation.** Turn J01–J20 into identified test cases - and the examples into checked docs. Run Rust writes → Go reads/queries - and Go writes → Rust reads/queries against PostgreSQL. Include negative - cases, exact number handling, null distinctions, multi-element correlated - containment, and batch ordering. Test SQL GIN/btree usage on representative - containment/range queries. Randomized ciphertexts need semantic parity; - deterministic fixtures are only for explicitly seeded test builds. - -The current wire order/leakage and extraction trust boundaries must be -documented accurately. This operation inventory does not claim a new -authenticated-query-completeness protocol or a storage-order leakage fix. -Stable blob framing is a separate deliverable tracked in -[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); it must -not be confused with the EQL JSON-tree storage format. +## Remaining work and release gates + +1. Capture the implemented wire grammar, errors, presence states and context + profile in the promised formal specification and conformance corpus. + General Rust definitions/caching and additional context roles can follow. +2. Close the explicit J14/J17 scope gaps. The supported all-match plaintext + route is subtree reads; a parent-bound child opener is separate engine + work. Do not add generic raw-entry decryption as a shortcut. +3. Replace the internal EQL serialize-to-Value/deserialize bridge with typed + conversion while preserving the frozen profile. This is an optimization, + not a missing Go JSON execution path. +4. Run the existing Rust/guest/Go/PostgreSQL suites on the release candidate, + mapping tests to J01–J20. Include typed/raw inputs, numeric/null cases, + both root policies, wildcard bounds, fallback, cascade tampering and known + omission limitations. Inspect real SQL parameter casts and index use. +5. Freeze post-cascade storage/search vectors and the matching EQL bundle. + Compare canonical projection semantics and cross-opening; ordinary + randomized bytes need not match. Replace temporary Go Vitamin C pins + before a final release. Opaque storage adoption is already done and is + a separate format from JSON-tree storage. + +This refresh is source and documentation review, not a fresh runtime test +run or a complete cryptographic audit. Final Rust example compilation, +package extraction and stronger zeroization/constant-time work remain later +API-plan activities; they must preserve the released Go/data contract. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md index 9c7d50e7c..09c7bd248 100644 --- a/docs/plans/2026-10-25-stack-encrypt-packaging.md +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -7,11 +7,12 @@ renames. Existing companion examples retain their working names until this proposal is accepted. The detailed JSON operation contract remains under separate user review. -**Sequencing update:** customer Go delivery supersedes the original -move-first sequence. The JSON stack headed by #1363 has merged. Vitamin C -#520 and #526 address storage and await review/release. Adopt those encodings -and deliver Go JSON/match search first. The package map below is the intended destination; neither a new -format crate nor public Rust facade completion is a customer release gate. +**Implementation update, 2026-10-27:** Go JSON and opaque storage have +landed on `next` through #1369/#1370/#1372/#1373/#1378/#1402; #1397 adds +short whole-word match terms. The customer path is remaining configuration +and conformance work plus release readiness, including replacing temporary +Vitamin C Go module pins. Rust stays on crates.io Vitamin C 0.8.0. Neither +crate extraction nor final Rust facade completion is a customer release gate. Source baseline: the current documentation branch and inspected local Vitamin C 0.8.0 sources. Stack's @@ -196,9 +197,16 @@ for the ciphertext wrappers; avoid an unchecked public string constructor. ## JSON names by role -Keep the existing EQL package and version modules. Avoid importing low-level -engine JSON documents into normal application examples alongside EQL output -types. Recommend this glossary for the first-class public surface: +Keep the existing EQL package and version modules. The merged public Go +surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery` and +`JSONEntry`, configured through `JsonSearch`; SQL uses `eql_v4_json_search`, +`query_json`, `query_json_leaf` and `eql_v4_json_entry`. Prepared Go objects +are `encrypt.JSONSelection` and `JSONRead`. + +The table below remains a proposed Rust facade glossary. Its `JsonTree*` +names describe handwritten Rust adapters, not the registered Go target. +Do not rename the working Go/SQL surface to this glossary as a customer +release prerequisite. Later Rust renames must preserve frozen dynamic IDs. | Working/current name | Recommended name | Role / home | | --- | --- | --- | @@ -235,13 +243,12 @@ or dynamic discriminants. ## Implementation order and acceptance -1. Use the merged JSON stack and specify Go-visible names/contracts and stable - profiles needed by the customer. Keep codec work in Vitamin C, coordinating - #519. Preserve distinct transport/storage semantics and freeze search - semantics as well as byte layouts. -2. Deliver and validate Go JSON, configurable match search and the selected - short-input solution under the customer-first sequence. Record frozen Go/data - fixtures; leave internal Rust plan ergonomics free to change. +1. Specify and validate the now-implemented Go JSON and opaque storage + contracts. Freeze post-cascade JSON/search semantics and exact fixtures. + Replace temporary Vitamin C Go module pins for a final release. +2. Complete outstanding Go match configuration and JSON read/conformance + gaps. Short whole-word matching is implemented in EQL; its leakage and + profile compatibility still require explicit release acceptance. 3. After delivery, establish the shared format crate and move the value model, traversal and dependent normalization/index adapters into Stack. Preserve frozen encodings and an acyclic graph; move tests with their code. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md index eeaf0afef..0ce88185e 100644 --- a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -26,6 +26,15 @@ source value requires explicit declaration, not inference from a name. ## Current implementation evidence +**Update, 2026-10-27:** #1379 makes open-root JSON verification load the +index keyset as well as the data key. `Pending` settlement and generic +execution entry points now require `DataKeySource + IndexKeySource`. +The future neutral provider split below must preserve that verification +requirement for JSON while allowing data-key-only capabilities where an +operation genuinely needs no index key. This change does not implement +plaintext-derived keyset routing or ZeroKMS lock-context plumbing. + + - [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts `IdentifiedBy` by UUID or name, resolves the keyset and its index key, and returns a bound `KeysetCipher`. The client has a bounded keyset cache