new(mdl-scan): flag scannable STOP-row patterns before the apply, warn-only until a field run (#349)

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- new(mdl-scan): **`bin/mdl-scan.sh` flags a script that carries a scannable STOP-row pattern, inside `exec.sh` before the lock, the snapshot and the apply — warn-only until a field run** ([#349](https://github.com/MendixMau/mxcli-project-toolkit/issues/349)). Four of the fourteen live STOP rows are text patterns with a captured failure, and each is one row of `bin/lib/mdl-scan-rules.tsv`: 1 (conditional Visible/Editable inside a customContent column, BUG-18 StorageLoadException), 12 (an expression in `ContentParams`, BUG-142 CE1613), 16 (a path inside `[%CurrentUser%]`, CE0161), 26 (a quoted association path in `Attribute:`, CE1613). Output is `STOP-<n> <file>:<line> <message> → <fix> [warn|block]`; exit 0 clean, 1 a `block` hit, 3 `warn` hits only, 2 usage. Every rule ships as `warn`: exec.sh prints the hits, notes them in the BUILD-LOG row and goes on. A rule moves to `block` after one field run where it caught a real failure and flagged nothing valid (candidates 12, 16, 26; STOP-1 stays `warn`, its evidence predates v0.24.0). The `block` path is kept and tested: it refuses before anything is touched, `FORCE_EXEC`/`SKIP_CHECK` do not skip it, and the only override is the user's `MXTK_SCAN_OVERRIDE="<reason>"`, printed and logged (#348 hardens it). The other rows need model state or judgement and stay prose; 1b, 4, 18 and 24 are scannable but have no captured failure in `bug-logs/`, so they are proposed, not shipped. `learned-mdl-preflight.md` marks the enforced rows with 🔎 (+6 baseline words), and row 16's example now uses generic entity names (`Decision_Partner`, `Account_Partner`) in place of project-specific ones; the verbatim form stays in the bug log. Fixture `tests/wave2/test-mdl-scan.sh` (52 checks: each rule fires on the bad MDL captured in its bug entry and is silent on the corrected form, keys and CE text asserted verbatim in the ledger, the independent review's three false-hit forms clean, warn and block exec paths). `tests/wave2/bench-mdl-scan.sh`: 4/9 captured failures flagged, 0/19 clean files flagged, 88–260 s of applies avoidable over the corpus. Field evidence: captured bug-log output only, no live run. — external reviewer feedback (2026-10-14), independent review of #355
 - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau
 - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync <remote>` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync <remote>` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote.<r>.push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau
 - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau

bin/lib/mdl-scan-rules.tsv

@@ -0,0 +1,24 @@
+# mdl-scan-rules.tsv — one row per STOP row of skills/learned-mdl-preflight.md that a text pattern
+# identifies. Read by bin/mdl-scan.sh. Columns, TAB-separated:
+#   id       STOP-<n>, the preflight row number
+#   kind     line          ERE on the comment-stripped, lowercased line (strings kept)
+#            customcontent ERE on the same line with string literals blanked, only inside a
+#                          datagrid column body whose header says ShowContentAs: customContent
+#            contentparams built-in: a ContentParams value that is not a bare attribute,
+#                          an unquoted association path or a quoted literal (pattern unused)
+#   severity warn   exec.sh prints the hit and goes on; the BUILD-LOG row notes it
+#            block  exec.sh refuses (MXTK_SCAN_OVERRIDE="<reason>" is the user's override)
+#            A rule ships as `warn`. It goes to `block` only after ONE field run in which it
+#            caught a real failure and flagged nothing valid, cited on the commit that flips
+#            it. Candidates: STOP-12, STOP-16, STOP-26. STOP-1 stays `warn`: its evidence
+#            predates v0.24.0, where `mx check` is 0 errors and Studio Pro load is untested.
+#   pattern  lowercase ERE (POSIX awk; no \b, no lookaround)
+#   message  what the hit is
+#   fix      the rewrite
+#   evidence the bug-log entry holding the captured failure; its fixture is
+#            tests/wave2/fixtures/mdl-scan/<id>.{bad,good}.mdl (`-- evidence:` lines)
+# A rule with no captured failure in bug-logs/ does not get a row (issue #349).
+STOP-1	customcontent	warn	(visible|editable)[[:space:]]*:[[:space:]]*\[	conditional Visible/Editable inside a datagrid customContent column (StorageLoadException on Studio Pro open)	set it in Studio Pro by hand, or bind dynamictext (Attribute: "State")	BUG-18
+STOP-12	contentparams	warn	-	ContentParams value is an expression; it is stored as an attribute path (CE1613 at mxbuild)	bind a bare attribute ({1} = Attr) or a quoted literal; make a computed label a String attribute	BUG-142
+STOP-16	line	warn	\[%currentuser/	path inside the [%CurrentUser%] token in an XPath constraint (CE0161)	keep the token bare and walk the whole path on the left: [Mod.A_B/Mod.B/Mod.B_Account = '[%CurrentUser%]']	bug-logs/mxcli-bugs.md "Any GRANT/REVOKE exec in a module fails once ANY entity there has an Account_Supplier-style cross-module XPath"
+STOP-26	line	warn	attribute[[:space:]]*:[[:space:]]*"[^"]*/[^"]*"	quoted association path in Attribute: is stored as one attribute named "Assoc/Attr" (CE1613)	write the path unquoted: Attribute: Assoc_A_B/Name	BUG-DRAFT-dg2-association-path-bindings

bin/mdl-scan.sh

@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# mdl-scan.sh — flag an MDL script that contains a STOP-row pattern, before anyone applies it.
+#
+#   bin/mdl-scan.sh <script.mdl> [-p <app.mpr>]
+#
+# One rule per row of skills/learned-mdl-preflight.md's STOP table that a text pattern in the
+# script identifies (issue #349). The rules are data: bin/lib/mdl-scan-rules.tsv, whose header
+# documents the columns and the three rule kinds. Rows that need model state or judgement are
+# not here; they stay prose in the skill, unmarked.
+#
+# Output: one line per hit, `STOP-<n> <file>:<line> <message> → <fix> [warn|block]`.
+# Exit:   0 clean · 1 at least one hit of a `block` rule · 3 hits, all of `warn` rules ·
+#         2 usage error (no script, unreadable file, bad flag, rules file missing).
+# Severity is a column of the rules file; its header says when a rule may go to `block`.
+# exec.sh refuses on 1 and prints the hits as warnings on 3.
+#
+# -p is accepted and checked, so a caller can pass the model the same way it passes it to
+# `mxcli check`. No current rule reads it: every row that needs model state stayed prose
+# (the PR for #349 lists them). A future rule that uses it must stay read-only.
+#
+# WHY. The STOP table is prose an agent reads, or skips. Every rule here was a script that
+# passed `mxcli check`, cost a full apply, and failed only at mxbuild or on Studio Pro open
+# (bug ids in the rules file). A text scan costs milliseconds; a failed apply costs 22-65 s
+# plus the restore (iterative-build-loop.md, "Gate cost").
+#
+# HOW IT READS A SCRIPT. Comments (`--`, `//`, `/* */`) and `$$ ... $$` bodies are removed
+# and the line is lowercased. `line` rules see string literals (STOP-16 lives inside one);
+# `customcontent` rules and all bracket counting see the line with literals blanked. A
+# customContent column's body is known by `{ }` depth, not indentation; a column with no body
+# is forgotten at the `)` that closes its header's scope or at the next `}`. A ContentParams
+# list ends at its matching `]` by depth, not at the last `]` on the line. It is a line
+# scanner, not a parser: a one-line column whose body opens and holds the conditional on the
+# header line itself is not seen as inside the column.
+#
+# Golden input: the bad MDL captured in bug-logs/mxcli-bugs.md for each rule, and the
+# StyleGallery example (applied, 0 errors in mx check) as the clean corpus —
+# tests/wave2/test-mdl-scan.sh, tests/wave2/bench-mdl-scan.sh. Plain bash 3.2 + POSIX awk.
+set -u
+
+usage() { echo "usage: bin/mdl-scan.sh <script.mdl> [-p <app.mpr>]" >&2; exit 2; }
+
+SCRIPT=""; MPR=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    -p) [ $# -ge 2 ] || usage; MPR="$2"; shift 2 ;;
+    -h|--help) sed -n '2,13p' "$0"; exit 0 ;;
+    -*) echo "mdl-scan: unknown option '$1'" >&2; usage ;;
+    *) [ -z "$SCRIPT" ] || usage; SCRIPT="$1"; shift ;;
+  esac
+done
+[ -n "$SCRIPT" ] || usage
+[ -f "$SCRIPT" ] && [ -r "$SCRIPT" ] || { echo "mdl-scan: cannot read script: $SCRIPT" >&2; exit 2; }
+if [ -n "$MPR" ] && [ ! -f "$MPR" ]; then echo "mdl-scan: -p: no such model: $MPR" >&2; exit 2; fi
+
+RULES="${MDL_SCAN_RULES:-$(cd "$(dirname "$0")" && pwd)/lib/mdl-scan-rules.tsv}"
+[ -r "$RULES" ] || { echo "mdl-scan: rules file missing: $RULES" >&2; exit 2; }
+
+awk -F'\t' -v file="$SCRIPT" '
+  FNR == NR {                                   # the rules file
+    if ($0 ~ /^#/ || NF < 6) next
+    n++; rid[n] = $1; rkind[n] = $2; rsev[n] = $3; rpat[n] = $4; rmsg[n] = $5; rfix[n] = $6
+    next
+  }
+  function hit(i, ln) {
+    printf "%s %s:%d %s → %s [%s]\n", rid[i], file, ln, rmsg[i], rfix[i], rsev[i]
+    hits++; if (rsev[i] == "block") blocks++
+  }
+  # strip(line): sets code (comments and $$-bodies removed, strings kept) and bare (strings
+  # blanked too). Same length, so an offset found in one cuts the other. inblock / indollar
+  # carry across lines.
+  function strip(line,    m, i, c, c2, instr) {
+    code = ""; bare = ""; m = length(line); i = 1; instr = 0
+    while (i <= m) {
+      c = substr(line, i, 1); c2 = substr(line, i, 2)
+      if (inblock)  { if (c2 == "*/") { inblock = 0; i += 2 } else i++; continue }
+      if (indollar) { if (c2 == "$$") { indollar = 0; i += 2 } else i++; continue }
+      if (instr) {
+        code = code c; bare = bare " "
+        if (c == "\047") instr = 0
+        i++; continue
+      }
+      if (c2 == "/*") { inblock = 1; i += 2; continue }
+      if (c2 == "$$") { indollar = 1; i += 2; continue }
+      if (c2 == "--" || c2 == "//") break
+      if (c == "\047") instr = 1
+      code = code c; bare = bare (instr ? " " : c)
+      i++
+    }
+    code = tolower(code); bare = tolower(bare)
+  }
+  # A ContentParams value is fine when it is a quoted literal, or a bare (optionally
+  # double-quoted) identifier, or an unquoted identifier path. Anything else is an expression.
+  function bad_value(v) {
+    sub(/^[[:space:]]+/, "", v); sub(/[[:space:],]+$/, "", v)
+    if (v ~ /^\047([^\047]|\047\047)*\047$/) return 0
+    if (v ~ /^"?[a-z_][a-z0-9_]*"?$/) return 0
+    if (v ~ /^[a-z_][a-z0-9_.]*(\/[a-z_][a-z0-9_.]*)*$/) return 0
+    return 1
+  }
+  function check_params(text, ln, i,    rest, v) {
+    rest = text
+    while (match(rest, /\{[0-9]+\}[[:space:]]*=/)) {
+      rest = substr(rest, RSTART + RLENGTH)
+      if (match(rest, /,[[:space:]]*\{[0-9]+\}[[:space:]]*=/)) v = substr(rest, 1, RSTART - 1)
+      else v = rest
+      if (bad_value(v)) { hit(i, ln); return }
+    }
+  }
+  # cp_feed(c, b): add one stretch of an open ContentParams list (code c, bare b) to cp_text,
+  # counting [ ] in b. Returns 1 when the list closes, cutting c at its matching ].
+  function cp_feed(c, b,    k, m, ch) {
+    m = length(b)
+    for (k = 1; k <= m; k++) {
+      ch = substr(b, k, 1)
+      if (ch == "[") cp_depth++
+      else if (ch == "]" && --cp_depth == 0) { cp_text = cp_text " " substr(c, 1, k - 1); return 1 }
+    }
+    cp_text = cp_text " " c; return 0
+  }
+  {
+    strip($0)
+    incc = (ccn > 0)                            # inside a customContent column body?
+    for (i = 1; i <= n; i++) {
+      if (rkind[i] == "line" && code ~ rpat[i]) hit(i, FNR)
+      else if (rkind[i] == "customcontent" && incc && bare ~ rpat[i]) hit(i, FNR)
+      else if (rkind[i] == "contentparams") {
+        if (cp_open) {                          # a ContentParams list still open from above
+          if (cp_feed(code, bare)) { check_params(cp_text, cp_line, i); cp_open = 0 }
+        } else if (match(bare, /contentparams[[:space:]]*:[[:space:]]*\[/)) {
+          k0 = RSTART + RLENGTH; cp_text = ""; cp_line = FNR; cp_depth = 1
+          if (cp_feed(substr(code, k0), substr(bare, k0))) check_params(cp_text, cp_line, i)
+          else cp_open = 1
+        }
+      }
+    }
+    if (bare ~ /^[[:space:]]*column[[:space:]]/) { pending = 0; colbase = pdepth }
+    if (bare ~ /showcontentas[[:space:]]*:[[:space:]]*customcontent/) pending = 1
+    m = length(bare)
+    for (j = 1; j <= m; j++) {
+      c = substr(bare, j, 1)
+      if (c == "(") pdepth++
+      else if (c == ")") { pdepth--; if (pending && pdepth < colbase) pending = 0 }  # header scope ended, no body
+      else if (c == "{") { depth++; if (pending) { cc[++ccn] = depth; pending = 0 } }
+      else if (c == "}") { pending = 0; if (ccn > 0 && cc[ccn] == depth) ccn--; depth-- }
+    }
+  }
+  END { exit (blocks > 0 ? 1 : (hits > 0 ? 3 : 0)) }
+' "$RULES" "$SCRIPT"