feat(go): encrypted JSON documents of Go types, with typed needles

languages/golang/CHANGELOG.md

@@ -12,6 +12,37 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Added
 
+- **Encrypted JSON documents of Go types** (#1351). A struct, map or slice
+  field tagged `encrypt_into=JsonSearch` is a document: the generated code
+  marshals it with `encoding/json` when it seals it, and reads the opened
+  document back into the declared type (`gensupport.GetDocument`,
+  `GetDocumentPtr`), so 64-bit integers and finite `float64`s round-trip
+  exactly. A pointer to one takes `null=sql`. The rest of a document comes
+  back as `encoding/json` reads it: invalid UTF-8 in a string becomes
+  U+FFFD, a nil map or slice is stored as JSON `null` (not `{}` or `[]`),
+  and a `time.Time` inside a document is stored as RFC 3339 text in its
+  own offset, which `EqAt` and the orderings compare as text. `stashgen` refuses, when it
+  runs, a type `encoding/json` cannot carry both ways: an interface such as
+  `any`, a struct field `encoding/json` drops (unexported, tagged
+  `json:"-"`, or two writing one member), a field tagged
+  `json:",string"` (stored as a JSON string, which a query by its number
+  does not match), an embedded pointer to an
+  unexported struct (written, never read back), a one-way `MarshalJSON` or
+  `MarshalText`, a channel, a function, a scalar, or a `[]byte`. The `Fields` entry's operands are Go
+  values now: `Contains`, `ContainedBy`, `EqAt`, `Lt`, `Lte`, `Gt`, `Gte`
+  and `Between` take `any`,
+  a `json.RawMessage` crossing as the JSON text it is and anything else
+  marshalled (`gensupport.JSONText`); a `[]byte`, a type defined over
+  one, or a pointer to either is refused, and so is a containment needle
+  that is a Go string, a pointer to one, or writes JSON `null`. A zero
+  struct field without `omitempty` narrows a containment query, so
+  `Contains` on a field whose type holds a struct refuses a needle of that
+  type (`gensupport.NewDocumentField`); the generated doc comments say how
+  to write a needle. `encrypt.DecodeJSON[T]` reads a value a selection or
+  subtree read opened into a Go type, numbers in an interface as
+  `json.Number`. An error from either direction names a Go type and never
+  the value.
+
 - **Encrypted JSON documents with `stashgen`** (#1351, #1352). A
   `json.RawMessage` field tagged `encrypt_into=JsonSearch` (options:
   `ordered=$.a;$.b[*].c`, `sealed_root=false`) is stored as an

languages/golang/cmd/stashgen/README.md

@@ -74,7 +74,7 @@ The first part of a tag is the field's name, which is the column name in a datab
 | `stash:"notes,encrypt"` | seal the field, with no index |
 | `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it |
 | `stash:"score,index=ore"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value |
-| `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` | on a `json.RawMessage`: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
+| `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` | on a `json.RawMessage`, or a struct, map or slice type: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
 | `stash:"attrs,index=json"` | refused: a JSON document is stored with `encrypt_into=JsonSearch`, not as an index beside a ciphertext |
 | `stash:"id,passthrough"` | store the field as it is |
 | `stash:"tenant,context_field"` | the field's value is the context of every other field in the struct, in place of a `context=` tag; stored as it is |
@@ -113,12 +113,55 @@ These words are the same as the Rust API's words for the same behaviour, with tw
 
 ### Encrypted JSON documents
 
-A `json.RawMessage` field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
-The field holds the document's JSON text; the engine parses it, so an object that names a member twice, an integer outside the 64-bit range and text that is not JSON are refused with `encrypt.ErrEncoding`.
-Values come back as JSON text holding the same values; key order and the spelling of numbers are not kept.
-A `*json.RawMessage` field is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind. A non-nil pointer to `null` is a stored JSON `null`.
+A field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
+Declare it as a struct, map or slice type, or as a `json.RawMessage` that holds the document's JSON text:
+
+```go
+type Profile struct {
+	Name   string  `json:"name"`
+	Age    int64   `json:"age"`
+	Visits []Visit `json:"visits"`
+}
+
+type Customer struct {
+	_       struct{}        `stash:"context=customers"`
+	ID      int64           `stash:"id,passthrough"`
+	Profile Profile         `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`
+	Notes   json.RawMessage `stash:"notes,encrypt_into=JsonSearch"`
+}
+
+// A needle for Contains: a field left nil is not in the query.
+type ProfileNeedle struct {
+	Name *string `json:"name,omitempty"`
+	Age  *int64  `json:"age,omitempty"`
+}
+
+q, err := customers.Fields.Profile.Contains(ctx, cipher, ProfileNeedle{Age: &age})
+older, err := customers.Fields.Profile.At.Age.Gt(ctx, cipher, 40)
+people, err := customers.Decrypt(ctx, cipher, rows) // each Profile is a Profile again
+```
+
+The generated code marshals a struct, map or slice type with `encoding/json` when it seals the field, and unmarshals the opened document into the declared type.
+It calls `encoding/json` directly, never through `map[string]any`, so an `int64`, a `uint64` and a finite `float64` come back exactly.
+The rest of the document comes back as `encoding/json` reads what it wrote, which is not always the value sealed:
+
+- Invalid UTF-8 in a Go string is written as U+FFFD, with no error: `"a\xffb"` comes back as `"a\uFFFDb"`, the replacement character in place of the byte. `stashgen` cannot refuse it, because it is a property of a value, not of a type, and finding it would need a reflective walk of each value at run time, which the generated code does not do. Check text that may not be UTF-8 with `utf8.ValidString` before you seal it.
+- A nil map or slice is stored as JSON `null`, and an empty one as `{}` or `[]`. Each reads back as it was (a field with `null=sql` stores a nil pointer as SQL `NULL` instead), but a query tells them apart: `Contains` with an empty map does not match a stored `null`.
+- A `time.Time` field is refused, since it is one value, not a document, but a `time.Time` inside a document is accepted. It is stored as the text its `MarshalJSON` writes: RFC 3339, in the time's own offset, with trailing zero nanoseconds left out. One instant in two zones is two strings, so `EqAt` with a time in another zone matches nothing, and an ordering on that path orders the text, which is not time order: `…T10:00:00+10:00` sorts after `…T01:00:00Z`, an earlier instant, and `…:00.5Z` sorts before `…:00Z`. If you query a time, store it as an integer (`t.UnixNano()`) or as UTC text of a fixed width. Any type with its own `MarshalJSON` or `MarshalText` is stored as the text it writes, and queried as that text.
+
+`stashgen` refuses a type that `encoding/json` cannot carry both ways: an interface such as `any` anywhere in it (it reads numbers back as `float64`); a struct field that `encoding/json` drops (unexported, tagged `json:"-"`, or one of two that write the same member); a `MarshalJSON` or `MarshalText` with no matching unmarshal method; a channel or a function.
+It refuses a field tagged `json:",string"` anywhere in a document: `encoding/json` stores its number or bool as a JSON string (`"age":"40"`), so `EqAt`, an ordering or a needle by the number would match nothing, and an ordered path would order the text. A document's numbers cross as their exact digits without it.
+An unexported embedded struct is fine: its fields are promoted and stored.
+An embedded pointer to an unexported struct is refused: `encoding/json` writes its fields but cannot read them back, since it cannot allocate the struct.
+An embedded pointer to an exported struct is accepted, but whether it was nil is not stored: it comes back non-nil when the document holds a member it promotes, and nil when it holds none, so a non-nil pointer whose promoted fields are all left out by `omitempty` comes back nil.
+It refuses a string, a number and a bool too: a document is a struct, map or slice.
+A `[]byte` is refused, since `encoding/json` writes it as base64: declare JSON text as `json.RawMessage`.
+The engine parses the JSON text, so an object that names a member twice and an integer outside the 64-bit range are refused with `encrypt.ErrEncoding`.
+A `json.RawMessage` comes back as JSON text that holds the same values; key order and the spelling of numbers are not kept.
+A pointer field (`*Profile`, `*json.RawMessage`) is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind.
+A nil pointer is SQL `NULL`; a non-nil `*json.RawMessage` holding `null` is a stored JSON `null`.
 A `json.RawMessage` sealed any other way (`encrypt`, `index=equality`) is bytes, as any type defined over `[]byte` is.
-`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): `stashgen` knows a document by the `json.RawMessage` type, where a scalar type is accepted with any type defined over it.
+`JsonSearch` takes `json.RawMessage` itself, or an alias of it (`type Doc = json.RawMessage`), and refuses a type defined over it (`type Doc json.RawMessage`): such a type has no `MarshalJSON`, so `encoding/json` would write it as a base64 string, not as the JSON text it holds, where a scalar type is accepted with any type defined over it.
 
 The type takes options in parentheses: `ordered=` names the paths whose numbers and strings carry an ordering term, separated by `;`, and `sealed_root=false` turns off the sealed root (on by default).
 `encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)` orders two paths.
@@ -150,7 +193,19 @@ SELECT id FROM customers ORDER BY (
 
 with `$1` from `At.VisitsAt.Selector(ctx, c)` and `$2` from `At.VisitsAt.Gt(ctx, c, v)`; for the latest visit, order by `ORDER BY 1 DESC NULLS LAST` inside and `DESC` outside.
 `OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
-The needle and the values are `json.RawMessage`: write the JSON you mean, such as `json.RawMessage(`+"`"+`{"active":true}`+"`"+`)`.
+A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
+`EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.
+A `[]byte`, a type defined over one, or a pointer to either, is refused rather than written as base64.
+A Go string, or a pointer to one, is refused as a `Contains` or `ContainedBy` needle, since it would be one JSON string: pass JSON text as `json.RawMessage`.
+So is a needle that writes JSON `null` (`nil`, or a nil pointer, map or slice), which only a null document contains.
+For `EqAt` and the orderings a Go string is the string value itself.
+A document contains a needle when it holds every member the needle writes, and `encoding/json` writes a struct field that has no `omitempty` even when it is zero.
+So a zero field narrows the query to the documents that hold that zero: `Profile{Name: "Alice"}` also asks for `"age": 0`.
+Give a needle struct pointer fields with `omitempty`, as `ProfileNeedle` above, or use a map.
+`Contains` refuses a needle of the field's own type when that type holds a struct, since that is the likeliest way to make the mistake; a type defined over it (`type wholeProfile Profile`) states that you mean every field.
+That includes a slice or map of structs: a `[]Visit` document refuses a `[]Visit` needle, since each `Visit` writes its zero fields; pass a `[]VisitNeedle`, a slice of a needle type with pointer fields and `omitempty`.
+`ContainedBy` takes the field's own type: there a zero field widens the query.
+Each value a selection or read opens is JSON text; `encrypt.DecodeJSON[T]` reads one into a Go type, with the same rules as a whole document.
 A position in a path is refused for `EqAt` and `Selector`, whose selector reads every position as the wildcard; read a position with `Subtree`.
 
 ### Optional fields