fix: search finds words in element documentation and microflow log messages

[stagknee]

Part of #1477: this PR is the indexing fix. The two flags from the issue are separate PRs: --fail-on-match in #1492, and --exclude-marketplace in #1493, which is stacked on this one.

What it fixes

A word that appears only in an element's documentation, or only in a microflow's log message, got No matches found. from mxcli search / SEARCH. The catalog table that holds the documentation finds it.

Documentation and a log message's StringTemplate.Text are plain strings, not Texts$Text, so the walk that fills the strings index never saw them.

How

• buildDocumentationStrings() in mdl/catalog/builder_strings.go is one INSERT … SELECT from the tables the builder has just filled, scoped to the snapshot being built:

  • the objects view;
  • attributes_data, microflow_parameters_data and java_action_parameters_data;
  • modules_data, for a module's domain-model documentation.

  Each row names its element (ObjectType such as ENTITY, ATTRIBUTE or CONSTANT) and its field (StringContext documentation).

  • Microflows and workflows are left out, because the typed extraction already indexes them; this avoids double rows.
  • Contract entities are left out, because a remote service writes that text.
  • The MODULE row is stored under the module's own name: CASE WHEN ObjectType = 'MODULE' THEN Name ELSE ModuleName END. The objects view gives it '' as ModuleName.

• extractLogNodeNames adds a log_message row beside the existing log_node one.

• The catalog schema goes from 24 to 25, so a cached catalog is rebuilt once.

• Docs:

  • docs-site/src/tools/search.md lists every documented element type.
  • The catalog-schema page, the SEARCH reference, the search help text and the syntax summary are updated.
  • The CHANGELOG has two entries under Fixed.

Evidence

• Tests: TestBuildStrings_IndexesElementDocumentation (including the module row's ModuleName), TestBuildStrings_UndocumentedElementsAddNoRows and TestExtractLogNodeNames_IndexesTheLogMessage.

• Control run: with buildDocumentationStrings and the log_message insert reverted, the first and third tests fail; with them restored, they pass. With only the MODULE CASE reverted, the first test fails on module: versus module:Sales. The full output is in the review reply below.

• Regression MDL: mdl-examples/bug-tests/1477-search-documentation.mdl contains a documented constant, entity and attribute, plus a microflow with a log message. After exec, this branch finds all four words and v0.26.0 finds none.

• Real app (Mendix 11.12.5), with a constant documented legacy rate source:

  $ mxcli search -p app.mpr "legacy rate"          # v0.26.0
  No matches found.
  
  $ mxcli search -p app.mpr "legacy rate"          # this branch
  String Matches (1)
  | QualifiedName | ObjectType | Match                          | StringContext | ModuleName |
  |---------------|------------|--------------------------------|---------------|------------|
  | Shop.RateUrl  | CONSTANT   | >>>legacy<<< >>>rate<<< source | documentation | Shop       |
  

  Building the catalog takes the same time as before, about 0.4–0.5 s.

• Finding: findings/mdl-other/2026-10-13-search-misses-words-only-in-documentation-or-log-message.json.

• Windows: the targeted tests pass. Across mdl/catalog, mdl/executor and cmd/mxcli, the failing tests are exactly the same set as on main.

• Linux (golang:1.26.6-bookworm, ANTLR 4.13.2, as in CI): make grammar build test check-mdl check-findings check-conformance check-migration-reference lint-go all pass.

🤖 Generated with Claude Code

[ako]

Thanks for this — the design is sound, and I checked the claims that carry it:

• Log messages: confirmed. LogMessageAction.MessageTemplate is a Microflows$StringTemplate, so the Texts$Text walk in buildTranslatableStrings never reaches it. The backend reader already wraps it as en_US, so textValue picks it up and the new log_message rows aren't duplicated.
• No double rows: excluding MICROFLOW/WORKFLOW avoids them. cachedMicroflows() returns microflows only, so nanoflow and rule documentation from objects isn't indexed twice.
• Strict mode: it can't turn "source index not built yet" into exit 2. The source FTS table is always created, just empty.
• Exit-code tests: they actually run under make test. They skip only on -short.
• Schema bump: going to 25 is needed for cached catalogs, and the history entry explains it.

One defect

--exclude-marketplace still lets a Marketplace module's own documentation through. The MODULE row in the objects view has '' as ModuleName, and buildDocumentationStrings copies ModuleName from it, so marketplaceModuleFilter (ModuleName not in (...)) never matches that row. The domain_model_documentation row is filtered correctly only because it uses Name. Suggested fix in the first SELECT:

SELECT QualifiedName, ObjectType, Description, 'documentation', '', Id,
       CASE WHEN ObjectType = 'MODULE' THEN Name ELSE ModuleName END
FROM objects ...

Please also add a documented Marketplace module to TestSearchWith_ExcludeMarketplace, so the test covers this case.

Smaller: the subquery in marketplaceModuleFilter reads modules without a snapshot filter. In a catalog holding more than one snapshot, a module that is Marketplace-sourced in one snapshot is excluded in all of them.

Against the contribution checklist (CLAUDE.md)

• Scope: this PR does four things. Documentation indexing and log_message are the bug, and --fail-on-match and --exclude-marketplace are two separate features. Could they be split, with the indexing fix and log_message in one PR and each flag on its own?
• Control run: please show TestBuildStrings_IndexesElementDocumentation and TestExtractLogNodeNames_IndexesTheLogMessage failing with buildDocumentationStrings and the log_message insert reverted, and paste that output here.
• Regression MDL: add mdl-examples/bug-tests/1477-search-documentation.mdl, reproducing the issue's documented-constant case, and confirm make check-mdl passes.
• Finding: add one file under .claude/skills/fix-issue/findings/mdl-catalog/ and confirm make check-findings passes. The insight worth recording: documentation and StringTemplate.Text are plain strings, so a walk keyed on Texts$Text can never see them.
• Real-app evidence: the commit says this was measured on a Mendix 11.12 project. Could you put that before/after mxcli search output in the description?

---

Generated by Claude Code

[stagknee]

Thanks for the careful review. All points are addressed. The PR is now split into three:

• fix: search finds words in element documentation and microflow log messages #1478 (this PR): indexing only, meaning element documentation plus the log_message fix. It has been force-pushed as a single commit on current main.
• feat: mxcli search --fail-on-match exits 1 on a hit, 0 on none #1492: --fail-on-match, branched from main, independent of this PR.
• feat: mxcli search --exclude-marketplace leaves out Marketplace modules #1493: --exclude-marketplace. It is stacked on this PR, because the case you found only exists once module documentation is indexed.

Defect: the MODULE row was missing its module name. It is fixed here as you suggested: CASE WHEN ObjectType = 'MODULE' THEN Name ELSE ModuleName END. TestBuildStrings_IndexesElementDocumentation now asserts ModuleName on the module's own documentation row, and it fails with only that CASE reverted (output below). #1493 adds the documented-Marketplace-module case to the exclude tests.

Snapshot scope of the Marketplace filter (in #1493): the modules lookup is now limited to the latest snapshot. It uses the same idiom as AddGraphAnalysis: SnapshotId = (select SnapshotId from snapshots order by rowid desc limit 1). TestSearchWith_ExcludeMarketplace_UsesTheCurrentSnapshot covers both directions with two snapshots. One limit remains: the strings/source FTS tables have no SnapshotId column, so the hits themselves can't be snapshot-scoped without a schema change. I left that alone.

Control run: the two tests fail with buildDocumentationStrings and the log_message insert reverted, and pass with them restored:

=== RUN   TestBuildStrings_IndexesElementDocumentation
    builder_strings_documentation_test.go:166: Sales.Home / documentation: documentation never reached the strings index
    builder_strings_documentation_test.go:166: Sales / documentation: documentation never reached the strings index
    builder_strings_documentation_test.go:166: Sales / domain_model_documentation: documentation never reached the strings index
    builder_strings_documentation_test.go:166: Sales.Order / documentation: documentation never reached the strings index
    builder_strings_documentation_test.go:166: Sales.Order.Total / documentation: documentation never reached the strings index
    builder_strings_documentation_test.go:166: Sales.Limit / documentation: documentation never reached the strings index
--- FAIL: TestBuildStrings_IndexesElementDocumentation (0.01s)
=== RUN   TestExtractLogNodeNames_IndexesTheLogMessage
    builder_strings_documentation_test.go:234: rows = [{objType:MICROFLOW value:'Orders' context:log_node elementID:act-log module:Sales}], want [{... context:log_node ...} {objType:MICROFLOW value:Order failed: zebracorn context:log_message elementID:act-log module:Sales}]
--- FAIL: TestExtractLogNodeNames_IndexesTheLogMessage (0.00s)
FAIL	github.com/mendixlabs/mxcli/mdl/catalog

(fix restored)
--- PASS: TestBuildStrings_IndexesElementDocumentation (0.02s)
--- PASS: TestExtractLogNodeNames_IndexesTheLogMessage (0.00s)
ok  	github.com/mendixlabs/mxcli/mdl/catalog

(only the MODULE CASE reverted)
    builder_strings_documentation_test.go:170: Sales / documentation: got {objectType:MODULE context:documentation elementID:mod-sales module:}, want {... module:Sales}
--- FAIL: TestBuildStrings_IndexesElementDocumentation (0.01s)

Regression MDL: mdl-examples/bug-tests/1477-search-documentation.mdl contains the documented constant from the issue, plus a documented entity, a documented attribute and a microflow with a log message. make check-mdl passes. After exec, this branch finds all four words; v0.26.0 finds none.

Finding: there's no mdl-catalog/ folder, so I followed the README's area mapping. The record is findings/mdl-other/2026-10-13-search-misses-words-only-in-documentation-or-log-message.json, with the insight you named: a Texts$Text walk can't see plain-string fields. make check-findings passes.

Real-app evidence: this is now in the description. It's a Mendix 11.12.5 app with a constant documented legacy rate source. v0.26.0 says No matches found., and this branch returns Shop.RateUrl | CONSTANT | documentation.

On Linux (golang:1.26.6-bookworm, ANTLR 4.13.2), each of the three branches passes make grammar build test check-mdl check-findings check-conformance check-migration-reference lint-go.

The two flag PRs both introduce SearchOptions, each with its own field. Whichever lands second will need a small rebase, and I'll do it.

[github-actions]

AI Code Review

Critical Issues

• None found.

Moderate Issues

• None found.

Minor Issues

• None found.

What Looks Good

• The PR comprehensively addresses issue search does not find words that appear only in an element's documentation (entities, attributes, constants, pages…), and has no exit code to gate on #1477 by making element documentation and microflow log messages searchable, adding --fail-on-match and --exclude-marketplace flags.
• Changes are properly scoped to the search functionality without modifying MDL syntax, avoiding full-stack wiring requirements.
• Test coverage is excellent: new unit tests for documentation indexing (builder_strings_documentation_test.go), log message extraction, search options, and fail-on-match behavior; plus an MDL example test (1477-search-documentation.mdl).
• Documentation updates are thorough: search command documentation, catalog schema documentation, skills, and changelog entries clearly explain the new behavior.
• Implementation follows existing patterns: documentation strings are built from already-populated catalog tables (no second model walk), schema version bumped appropriately, and SQL uses parameter binding for safety.
• Code quality is high: focused changes, clear comments, and consistent with project conventions (e.g., excluding microflows/workflows to avoid duplication with typed extraction).

Recommendation

Approve the PR. The changes are well-tested, documented, and maintain backward compatibility while extending search functionality as requested. No blocking issues identified.

---

Automated review via OpenRouter (Nemotron Super 120B) — workflow source

[ako]

All points from the first round are addressed: the split into #1492 / #1493, the module-row ModuleName fix with a test that now checks it, the regression MDL (parses: ✓ Syntax OK (4 statements)), the finding, and the control and real-app before/after in the description. The search tables having no SnapshotId was already the case before this PR, so it's fine as a follow-up.

Optional: the hand-written list of covered types in search --help will drift as the objects view grows. "the documentation of every element the catalog records" would stay true.

LGTM once CI is green.

---

Generated by Claude Code