You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
match index: "Mike Ng" no longer finds "Mike Nguyen" with short words; cut query word tokens only when the query has no n-gram #1398
Stack Encrypt's match index (the encrypted full-text index in packages/stack-encrypt/src/sem) cuts text into 3-character pieces ("trigrams"), hashes each under a secret key, and sets bits in a fixed-size bit array (a Bloom filter). A row matches a query when it holds every bit the query sets. With the short_words option (#1397, which EQL v4's match types turn on), every two-letter word is also one token of its own, so "Ng" is searchable.
Problem
The query side cuts word tokens too, whenever the query holds a two-letter word. So a query's two-letter words must be whole words of the row, and queries that found rows through their trigrams alone now miss:
@@ 'Mike Ng' no longer finds "Mike Nguyen" or "Mike Ngo".
@@ 'san fr' no longer finds "San Francisco".
@@ 'ke Ng' no longer finds "Mike Ng".
This breaks typeahead and partial-name search on EQL v4 text columns. A query of a two-letter word alone is fine: before short_words it was refused, so it found nothing.
Proposal
Cut a query's word tokens only when the query cuts no n-gram. "Ng" stays one word token and finds "Ng" and "Mike Ng"; "Mike Ng" is its trigrams alone and finds "Mike Nguyen" again, as it did before short_words.
The stored side does not change (a stored value keeps its word tokens), so no stored filter moves and no re-encryption is needed.
Every query derivation follows the one rule: the typed query chain (which today derives a query's filter through the index's stored operation), the data-plan term/query_term path, eql-bindings' TextMatchQuery and TextSearchMatchQuery, and the verifier (QueryTokens), so a filter and its verify agree.
Update the docs that say a query's two-letter words must be whole words (MatchOptions::short_words, the CHANGELOG, the eql-bindings and Go READMEs).
Relationship to other work
Raised in review of #1397 (thread on tokenize.rs), where the owner preferred this rule. Sibling of the character-filter follow-up for word separators.
Activity