From dccf72aeaae8e347bc525e6c4dc74d34258ffd5c Mon Sep 17 00:00:00 2001 From: stagknee <241821830+stagknee@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:52:57 -0500 Subject: [PATCH 1/2] feat: mxcli search --exclude-marketplace leaves out Marketplace modules (#1477) An audit of the app's own text reported a vendor module's text beside the project's. The flag drops hits whose module was downloaded from the Marketplace, a module's own documentation included. It is off by default and the SEARCH statement is unchanged. Which modules are Marketplace modules is read from the current snapshot only, so a module that was one in an older snapshot is not left out of every later search. Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 1 + cmd/mxcli/cmd_query.go | 8 + cmd/mxcli/main.go | 1 + cmd/mxcli/syntax/features_misc.go | 5 +- docs-site/src/reference/query/search.md | 2 +- docs-site/src/tools/search.md | 7 + mdl/executor/cmd_catalog.go | 37 +++-- .../cmd_search_exclude_marketplace_test.go | 149 ++++++++++++++++++ 8 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mdl/executor/cmd_search_exclude_marketplace_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 42f7011b9..82ac4df7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ Headline: **`exec` is all or nothing: when a statement fails at write time, the ### Added - **`mxcli search --fail-on-match` turns the search into a gate** — a pipeline that must refuse a change when a forbidden word turns up in the project's text had no exit status to read: `search` exited 0 whether it found something or not. With the flag it exits 1 when there is at least one hit and 0 when there is none, printing the results as before (a JSON array under `--json`). A search that cannot run, such as an unbalanced `"` or an unknown project, exits 2 so that 1 always means a match; the `SEARCH` statement still answers an unrunnable query with "No matches found.". Without the flag a hit exits 0 as before. Rules: docs-site "Full-Text Search". +- **`mxcli search --exclude-marketplace` leaves out hits inside Marketplace modules** (mendixlabs/mxcli#1477) — an audit of the app's own text (a leftover `TODO`, a word that must not ship) reported a vendor module's text beside the project's own. The flag drops hits whose module was downloaded from the Marketplace, including the module's own documentation and domain model documentation; it is off by default, because `search` is read-only and the default must not change what existing callers see (`layout` skips Marketplace modules because it writes). A module counts as a Marketplace module by the current snapshot of the catalog only, so one that was a Marketplace module in an older snapshot is not left out of every later search. The `SEARCH` statement is unchanged. Rules: docs-site "Full-Text Search". - **`filemanager` — the classic file manager (`Forms$FileManager`) in MDL** (mendixlabs/mxcli#151) — `filemanager fileManager1 (Label: 'File', Type: Upload, MaxFileSize: 10, AllowedExtensions: '.xlsx', ShowFileInBrowser: false)` inside a data view on a `System.FileDocument` specialization. Before, there was no way to author one: `pluggablewidget 'Forms$FileManager'` had no definition, the bare keyword was refused as MDL-WIDGET25, and `describe page` printed a stored one as a NOT re-executable comment, so describe → exec dropped it. Omitted properties take Studio Pro's defaults (`Both`, 5 MB, any file, false); the written BSON has the property set of a Studio Pro-authored file manager. `alter page … set` changes all four properties. A file manager in no data container is MDL-WIDGET45 at `check`, and one in a data view on an entity that is not a FileDocument is refused by `exec` — both were mxbuild's CE6810 after a clean `exec`. - **A module's domain model as SVG, in the companion and on the command line** — `mxcli describe --format svg domainmodel ` writes a finished SVG: entities at the positions the model stores, coloured by kind, with attributes, associations, multiplicities and generalizations. Boxes are sized to fit every name, pulling the stored layout apart just enough instead of clipping. In the Claude Code companion, selecting a module's Domain Model node in the desktop app or VS Code shows it at actual size, with pan, Fit and a full-width view; the terminal lists the entities as before. - **`Visible: roles (Mod.A, …)` — role-based widget visibility** (mendixlabs/mxcli#1460) — Studio Pro's "visible for module roles" was never read by `describe`, had no MDL spelling, was written empty and was dropped by `alter … set Visible`. It is now read, written and combinable with an expression or attribute condition, on widgets, tab pages and navigation list items. A tab page's (mendixlabs/mxcli#1465) and a layout grid row's `Visible:` (mendixlabs/mxcli#1461) are now written too; `describe` → `exec` used to make a Studio Pro row always visible. diff --git a/cmd/mxcli/cmd_query.go b/cmd/mxcli/cmd_query.go index 140469341..d1335b404 100644 --- a/cmd/mxcli/cmd_query.go +++ b/cmd/mxcli/cmd_query.go @@ -263,12 +263,18 @@ Gating on a word: index cannot run, or any other error, exits 2 so that 1 always means "something matched". +Marketplace modules: + --exclude-marketplace + leave out hits inside modules downloaded from the + Marketplace. Off by default: every module is searched. + Examples: mxcli search -p app.mpr "validation" mxcli search -p app.mpr "Customer" --json | jq '.[].qualifiedName' mxcli search -p app.mpr "Customer" --format names mxcli search -p app.mpr "error" -q --format names | xargs -I {} mxcli describe -p app.mpr microflow {} mxcli search -p app.mpr "TODO" --fail-on-match + mxcli search -p app.mpr "TODO" --exclude-marketplace `, Args: cobra.ExactArgs(1), Run: func(cmd *cobra.Command, args []string) { @@ -276,6 +282,7 @@ Examples: format := resolveFormat(cmd, "table") quiet, _ := cmd.Flags().GetBool("quiet") failOnMatch, _ := cmd.Flags().GetBool("fail-on-match") + excludeMarketplace, _ := cmd.Flags().GetBool("exclude-marketplace") // Under --fail-on-match, exit status 1 means "something matched", so a // search that could not run says so with 2 rather than with 1. @@ -315,6 +322,7 @@ Examples: // Execute search with format option query := strings.ReplaceAll(args[0], "'", "''") hits, err := exec.SearchWith(query, format, executor.SearchOptions{ + ExcludeMarketplace: excludeMarketplace, // A gate must not read "the index refused the query" as "no match". Strict: failOnMatch, }) diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 276952725..7221997c9 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -391,6 +391,7 @@ func init() { searchCmd.Flags().StringP("format", "f", "table", "Output format: table, names (for JSON use --json; \"json\" is accepted as a deprecated alias)") searchCmd.Flags().BoolP("quiet", "q", false, "Suppress connection and status messages (for piping)") searchCmd.Flags().Bool("fail-on-match", false, "Exit 1 when there is at least one hit, 0 when there is none (an error exits 2)") + searchCmd.Flags().Bool("exclude-marketplace", false, "Leave out hits inside modules downloaded from the Marketplace") // Callers/callees command flags callersCmd.Flags().BoolP("transitive", "t", false, "Find transitive (indirect) callers") diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index c6d0cef2f..43a59bc65 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -1186,7 +1186,7 @@ mxcli -p app.mpr --json -c "LIST FOLDERS IN MyModule" Syntax: `SEARCH ''; -- CLI -mxcli search -p app.mpr "" [--format table|names|json] [-q] [--fail-on-match] +mxcli search -p app.mpr "" [--format table|names|json] [-q] [--fail-on-match] [--exclude-marketplace] -- Raw FTS queries SELECT * FROM CATALOG.STRINGS WHERE strings MATCH ''; @@ -1200,6 +1200,9 @@ mxcli search -p app.mpr "validation" -q --format names -- Gate: exit 1 on a hit, 0 on none (2 when the search cannot run) mxcli search -p app.mpr "TODO" --fail-on-match +-- Leave out hits inside Marketplace modules +mxcli search -p app.mpr "TODO" --exclude-marketplace + -- FTS5 operators SEARCH 'word1 OR word2'; SEARCH '"exact phrase"'; diff --git a/docs-site/src/reference/query/search.md b/docs-site/src/reference/query/search.md index 1e380626d..b507bb4f2 100644 --- a/docs-site/src/reference/query/search.md +++ b/docs-site/src/reference/query/search.md @@ -8,7 +8,7 @@ Performs a full-text search across all strings, messages, captions, the documentation of project elements (entities, attributes, constants, pages and more), and MDL source in the project. Uses FTS5 (SQLite full-text search) for fast matching. Returns results grouped by document type and location. -This command requires the catalog to be populated. If search returns no results, run `REFRESH CATALOG FULL` first. The search is also available as a CLI subcommand: `mxcli search -p app.mpr "keyword" --format names|json`. +This command requires the catalog to be populated. If search returns no results, run `REFRESH CATALOG FULL` first. The search is also available as a CLI subcommand: `mxcli search -p app.mpr "keyword" --format names|json`. The CLI adds `--exclude-marketplace`; see [Full-Text Search](../../tools/search.md). ## Parameters diff --git a/docs-site/src/tools/search.md b/docs-site/src/tools/search.md index 03fc035ff..390b64487 100644 --- a/docs-site/src/tools/search.md +++ b/docs-site/src/tools/search.md @@ -41,6 +41,9 @@ mxcli search -p app.mpr "validation" --json # Exit 1 when anything matches (0 when nothing does) mxcli search -p app.mpr "validation" --fail-on-match + +# Leave out Marketplace modules +mxcli search -p app.mpr "validation" --exclude-marketplace ``` ## Output Formats @@ -122,6 +125,10 @@ mxcli search -p app.mpr "TODO" --fail-on-match --json Without the flag a hit exits `0`, as before. +## Marketplace modules + +By default every module is searched, Marketplace modules included, so existing results do not change. `--exclude-marketplace` leaves out hits inside modules downloaded from the Marketplace, which is what an audit of the app's own text usually wants: the vendor's documentation is not yours to edit. A Marketplace module's own documentation and domain model documentation are left out with the rest of it. + ## Comparison with Catalog Queries `SEARCH` provides quick keyword-based search, while catalog queries (`SELECT FROM CATALOG.*`) provide structured SQL-based queries with filtering, joining, and aggregation. Use `SEARCH` for quick discovery and catalog queries for precise analysis. diff --git a/mdl/executor/cmd_catalog.go b/mdl/executor/cmd_catalog.go index e00e7b9ab..8c765ca04 100644 --- a/mdl/executor/cmd_catalog.go +++ b/mdl/executor/cmd_catalog.go @@ -1091,9 +1091,12 @@ func preWarmCache(ctx *ExecContext) { describeDefaultLanguage(ctx) } -// SearchOptions harden a search. The zero value is what the SEARCH statement -// uses. +// SearchOptions narrow or harden a search. The zero value is what the SEARCH +// statement uses. type SearchOptions struct { + // ExcludeMarketplace drops hits in modules downloaded from the Marketplace, + // so an audit of the app's own text does not report a vendor's. + ExcludeMarketplace bool // Strict returns the index's error for a query it cannot run (an FTS5 syntax // error, say) instead of reporting "no matches". A caller that gates on the // absence of hits needs the difference: "nothing found" and "could not look" @@ -1101,15 +1104,31 @@ type SearchOptions struct { Strict bool } +// marketplaceModuleFilter is the WHERE fragment that keeps a search to modules +// the project owns. A module's Source is empty unless it came from the +// Marketplace (see the modules table). +// +// Only the current snapshot's modules count: the modules view holds a row per +// snapshot, and a module that came from the Marketplace in an older one would +// otherwise be left out of every later search. The current snapshot is the +// latest one recorded, as AddGraphAnalysis reads it. +const marketplaceModuleFilter = " and ModuleName not in (select Name from modules" + + " where COALESCE(Source, '') <> ''" + + " and SnapshotId = (select SnapshotId from snapshots order by rowid desc limit 1))" + // searchSQL returns the strings-index and source-index queries for a search. -func searchSQL(query string) (stringsSQL, sourceSQL string) { +func searchSQL(query string, opts SearchOptions) (stringsSQL, sourceSQL string) { + filter := "" + if opts.ExcludeMarketplace { + filter = marketplaceModuleFilter + } q := escapeFTSQuery(query) stringsSQL = fmt.Sprintf( - "select QualifiedName, ObjectType, snippet(strings, 2, '>>>', '<<<', '...', 32) as Match, StringContext, ModuleName from strings where strings match '%s' limit 50", - q) + "select QualifiedName, ObjectType, snippet(strings, 2, '>>>', '<<<', '...', 32) as Match, StringContext, ModuleName from strings where strings match '%s'%s limit 50", + q, filter) sourceSQL = fmt.Sprintf( - "select QualifiedName, ObjectType, snippet(source, 2, '>>>', '<<<', '...', 48) as Match, ModuleName from source where source match '%s' limit 50", - q) + "select QualifiedName, ObjectType, snippet(source, 2, '>>>', '<<<', '...', 48) as Match, ModuleName from source where source match '%s'%s limit 50", + q, filter) return stringsSQL, sourceSQL } @@ -1145,7 +1164,7 @@ func searchTable(ctx *ExecContext, query string, opts SearchOptions) (int, error return 0, err } - stringsSQL, sourceSQL := searchSQL(query) + stringsSQL, sourceSQL := searchSQL(query, opts) hits := 0 // Search strings table @@ -1261,7 +1280,7 @@ func searchWith(ctx *ExecContext, query, format string, opts SearchOptions) (int var allResults []searchResult seen := make(map[string]bool) // For deduplicating names format - stringsSQL, sourceSQL := searchSQL(query) + stringsSQL, sourceSQL := searchSQL(query, opts) // Search strings table strResult, err := runSearchQuery(ctx, stringsSQL, opts) diff --git a/mdl/executor/cmd_search_exclude_marketplace_test.go b/mdl/executor/cmd_search_exclude_marketplace_test.go new file mode 100644 index 000000000..5e12460b1 --- /dev/null +++ b/mdl/executor/cmd_search_exclude_marketplace_test.go @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/catalog" +) + +// moduleSeed is one row of the modules table: which snapshot it belongs to, the +// module, and its Source (empty unless the module came from the Marketplace). +type moduleSeed struct{ snapshot, name, source string } + +// seedSearchCatalog writes snapshots (oldest first, so the last one is the +// current snapshot), the given module rows and a documentation hit in each of +// Sales, Vendor and the module rows themselves, as the catalog builder writes +// them: a module's own documentation carries the module's name in ModuleName. +func seedSearchCatalog(t *testing.T, snapshots []string, modules []moduleSeed) func(catalog.CatalogDB) { + return func(db catalog.CatalogDB) { + t.Helper() + var stmts []string + for _, s := range snapshots { + stmts = append(stmts, `INSERT INTO snapshots (SnapshotId, ProjectId) VALUES ('`+s+`', 'p')`) + } + for i, m := range modules { + stmts = append(stmts, `INSERT INTO modules_data (Id, Name, QualifiedName, Source, SnapshotId, ProjectId) + VALUES ('m`+string(rune('a'+i))+`', '`+m.name+`', '`+m.name+`', '`+m.source+`', '`+m.snapshot+`', 'p')`) + } + stmts = append(stmts, + `INSERT INTO strings (QualifiedName, ObjectType, StringValue, StringContext, ModuleName) + VALUES ('Sales.Limit', 'CONSTANT', 'Upper bound for the quuxwidget', 'documentation', 'Sales')`, + `INSERT INTO strings (QualifiedName, ObjectType, StringValue, StringContext, ModuleName) + VALUES ('Vendor.Limit', 'CONSTANT', 'Upper bound for the quuxwidget', 'documentation', 'Vendor')`, + // A module's own documentation: the objects view holds '' as the + // module's ModuleName, and the builder stores the module's Name instead. + `INSERT INTO strings (QualifiedName, ObjectType, StringValue, StringContext, ModuleName) + VALUES ('Vendor', 'MODULE', 'Vendor notes on the quuxwidget', 'documentation', 'Vendor')`, + `INSERT INTO strings (QualifiedName, ObjectType, StringValue, StringContext, ModuleName) + VALUES ('Vendor', 'MODULE', 'Vendor domain notes on the quuxwidget', 'domain_model_documentation', 'Vendor')`, + ) + for _, q := range stmts { + if _, err := db.Exec(q); err != nil { + t.Fatalf("seed: %v\n%s", err, q) + } + } + } +} + +// searchHits runs a JSON search and returns "Qualified.Name/context" per hit. +func searchHits(t *testing.T, seed func(catalog.CatalogDB), opts SearchOptions) []string { + t.Helper() + // One fixture per search: a second search on the same context made the + // TempDir cleanup fail on Windows (catalog.db still in use). + ctx, out, _ := searchCacheFixtureSeeded(t, "source", nil, seed) + if _, err := searchWith(ctx, "quuxwidget", "json", opts); err != nil { + t.Fatal(err) + } + var rows []struct { + QualifiedName string `json:"qualifiedName"` + StringContext string `json:"stringContext"` + } + if err := json.Unmarshal(out.Bytes(), &rows); err != nil { + t.Fatalf("not JSON: %v\n%s", err, out.String()) + } + var hits []string + for _, r := range rows { + hits = append(hits, r.QualifiedName+"/"+r.StringContext) + } + return hits +} + +func sameSet(t *testing.T, got, want []string) { + t.Helper() + if len(got) != len(want) { + t.Fatalf("hits = %v, want %v", got, want) + } + seen := map[string]bool{} + for _, g := range got { + seen[g] = true + } + for _, w := range want { + if !seen[w] { + t.Fatalf("hits = %v, want %v", got, want) + } + } +} + +var sales = moduleSeed{"s1", "Sales", ""} +var vendor = moduleSeed{"s1", "Vendor", "Marketplace v1.0.0"} + +// The default keeps every module, as it always has; the option drops a +// Marketplace module's hits and nothing else - the module's own documentation +// and domain model documentation included, which a filter on ModuleName missed +// while the builder stored an empty one for a module. +func TestSearchWith_ExcludeMarketplace(t *testing.T) { + seed := seedSearchCatalog(t, []string{"s1"}, []moduleSeed{sales, vendor}) + + sameSet(t, searchHits(t, seed, SearchOptions{}), []string{ + "Sales.Limit/documentation", "Vendor.Limit/documentation", + "Vendor/documentation", "Vendor/domain_model_documentation", + }) + sameSet(t, searchHits(t, seed, SearchOptions{ExcludeMarketplace: true}), []string{ + "Sales.Limit/documentation", + }) +} + +// The same option reaches the table and names formats, not only JSON. +func TestSearchWith_ExcludeMarketplace_AllFormats(t *testing.T) { + seed := seedSearchCatalog(t, []string{"s1"}, []moduleSeed{sales, vendor}) + for _, format := range []string{"table", "names"} { + t.Run(format, func(t *testing.T) { + ctx, out, _ := searchCacheFixtureSeeded(t, "source", nil, seed) + if _, err := searchWith(ctx, "quuxwidget", format, SearchOptions{ExcludeMarketplace: true}); err != nil { + t.Fatal(err) + } + got := out.String() + if !strings.Contains(got, "Sales.Limit") || strings.Contains(got, "Vendor") { + t.Errorf("want Sales.Limit only, got:\n%s", got) + } + }) + } +} + +// The catalog can hold more than one snapshot. Only the current one (the latest +// recorded) says which modules are Marketplace modules: a module that was one in +// an older snapshot is the project's own now, and the other way round. +func TestSearchWith_ExcludeMarketplace_UsesTheCurrentSnapshot(t *testing.T) { + // Vendor came from the Marketplace in s1 and is the project's own in s2. + seed := seedSearchCatalog(t, []string{"s1", "s2"}, []moduleSeed{ + sales, vendor, + {"s2", "Sales", ""}, {"s2", "Vendor", ""}, + }) + sameSet(t, searchHits(t, seed, SearchOptions{ExcludeMarketplace: true}), []string{ + "Sales.Limit/documentation", "Vendor.Limit/documentation", + "Vendor/documentation", "Vendor/domain_model_documentation", + }) + + // Vendor is a Marketplace module in the current snapshot only. + seed = seedSearchCatalog(t, []string{"s1", "s2"}, []moduleSeed{ + sales, {"s1", "Vendor", ""}, + {"s2", "Sales", ""}, {"s2", "Vendor", "Marketplace v2.0.0"}, + }) + sameSet(t, searchHits(t, seed, SearchOptions{ExcludeMarketplace: true}), []string{ + "Sales.Limit/documentation", + }) +} From e425fa4ba756e0d1219a4ec00a4a30b409716dba Mon Sep 17 00:00:00 2001 From: stagknee <241821830+stagknee@users.noreply.github.com> Date: Wed, 14 Oct 2026 17:12:31 -0500 Subject: [PATCH 2/2] docs: search --help describes the documented elements by rule, not by a list The hand-written list of document types would drift as the objects view grows; "every element the catalog records" stays true. Suggested in review of #1478. Co-Authored-By: Claude Opus 5.5 --- cmd/mxcli/cmd_query.go | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/cmd/mxcli/cmd_query.go b/cmd/mxcli/cmd_query.go index d1335b404..551127458 100644 --- a/cmd/mxcli/cmd_query.go +++ b/cmd/mxcli/cmd_query.go @@ -231,18 +231,11 @@ Searches across string literals (captions, labels, messages), the documentation of project elements, and MDL source definitions. Requires at least a FULL catalog build (done automatically). -What is searched, besides captions, labels and messages, is the documentation of: - entities; attributes; associations; enumerations; constants; pages, snippets, - layouts, building blocks and page templates; microflows, nanoflows, rules and - workflows; Java and JavaScript actions; microflow and Java action parameters; - consumed and published REST services and business event services; import and - export mappings, JSON structures and data transformers; scheduled events, task - queues, regular expressions, image and icon collections and menus; agents, AI - models, knowledge bases and consumed MCP services; and a module's domain model - documentation. That is every document type whose catalog table carries a - documentation text. - Not searched: documentation of an enumeration value (Mendix has no such - field) and the summary text of a remote service's contract. +Besides captions, labels and messages, it searches the documentation of every +element the catalog records (entities, attributes, constants, pages, microflows, +Java actions, REST services, ... and a module's domain model documentation). +Not searched: the documentation of an enumeration value (Mendix has no such +field) and the summary text of a remote service's contract. A hit names the element (qualified name, type) and the field: the context column says "documentation" for a documentation hit.