match index: "Ng." and "Ng-Smith" hold no word "ng"; add a character filter to the text pipeline #1399

Background

Stack Encrypt's match index (packages/stack-encrypt/src/sem) normalizes text (Unicode NFC), lower-cases it, and cuts it into tokens: n-grams of the whole text, or words under the standard tokenizer, and with the short_words option (#1397) each two-letter word whole. Words are split on a fixed set: space, comma, semicolon, colon and exclamation mark.

Problem

There is no character filter in the text pipeline, so punctuation and other whitespace stay inside words:

• "Mike Ng." holds the word ng. (three characters), so it gets no ng word token and the query "Ng" does not find it.
• "Ng-Smith", "O'Ng", "(Ng)" and "Ng\nAnna" likewise carry no ng token.

Names and free text routinely carry full stops, hyphens, apostrophes, brackets, tabs and newlines, so two-letter words next to them are not searchable as words.

Proposal

1. Add a character filter stage to the match index's text pipeline, run before tokenizing, that maps punctuation and all Unicode whitespace to a word separator (an option, part of the index's identity like the others, so existing indexes keep their tokens).
2. Decide whether EQL v4's match types turn it on (EQL v4 is unreleased, so this can still be part of its format).
3. Tests for each character class above, for both tokenizers and for short words.

Relationship to other work

Raised in review of #1397 (thread on tokenize.rs), where the owner asked for it as a follow-up. Sibling of the query-side word-token follow-up.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu