diff --git a/CHANGELOG.md b/CHANGELOG.md index 20342404..64fe5e10 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- fix(graph-sweep, app-facts): **the catalog guard checks the refs table, not the `build_mode` label; mxcli v0.24+ labels a complete catalog `source` and both scripts refused every one** ([#259](https://github.com/MendixMau/mxcli-project-toolkit/issues/259)). Field report: four module reviews recorded the sweep as "Instrument FAULT" on a fresh catalog with 5,368 refs rows. Now only an empty or kind-less refs table FAULTs, naming the mode and the per-kind edge counts; a model with no microflow-to-microflow calls (reached only from pages and buttons) still sweeps, and graph-sweep warns when no orphan-read kind is present. `manifest.json` records the real mode. `tests/wave2/test-catalog-edge-guard.sh` (20 checks, both layouts) fails 16 on master's scripts, including the verbatim FAULT line. Synthetic catalogs; a field run on a v0.24 project is still owed. — AlvinnYeo +- fix(conformance-check, review-module, verify-module, coverage-ledger): **per-BRD ledgers (`/coverage-ledger/.md`) are all measured, one report per BRD, review/verify read every report of the run and accept a directory-only ledger, and an index file is named as one instead of "0 measurable rows"** ([#260](https://github.com/MendixMau/mxcli-project-toolkit/issues/260)). Field report: every module with an index ended in `FAULT: 0 measurable rows extracted from 1 ledger(s).`, while `LEDGER_FILE` on one per-BRD file measured 128 rows. `LEDGER_FILE` (file or directory) is in the usage text. **Upgrade notes:** (a) a `LEDGER_FILE` that does not exist now FAULTs instead of silently falling back to another ledger, and `verify-module.sh` reads `LEDGER_FILE` and passes it on to conformance through the environment, so a stale export now FAULTs the conformance rung; (b) baseline pointers of per-BRD rows are re-keyed `:`, so a project that had both a filled index and a per-BRD directory sees its old rows go unmatched and the new ones re-seeded as "newly baselined" on the first run after upgrade, with no regressions reported: run conformance once, then re-check. Per-BRD reports reach `docs/report.json` via #310, whose `report-normalize.js` filter is extended there (it reads only `report-.tsv` today). Fixtures: `test-conformance-per-brd-ledger.sh` (19, both layouts), `test-review-conformance-per-brd.sh` (6, both layouts), `test-verify-ledger-hint.sh` (4), all failing on master. A field run on a per-BRD project is still owed. — AlvinnYeo +- fix(verify-module): **a stack rung exit 4 from `test-stack-up.sh` (port held by another process) is an instrument FAULT, not a FINDING** — follow-up to [#313](https://github.com/MendixMau/mxcli-project-toolkit/pull/313), whose header says to record it as blocked, not as a test failure; review of #313 found verify-module grading it as findings. Scoped to the stack rung (exit 4 is NOT APPLICABLE from conformance-check). `tests/wave2/test-verify-stack-blocked.sh` (6) fails 2 on master. — AlvinnYeo (port-holder report), review of #313 - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run : N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst - fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo diff --git a/agents/review-agent.md b/agents/review-agent.md index b925be50..68d25615 100644 --- a/agents/review-agent.md +++ b/agents/review-agent.md @@ -141,9 +141,10 @@ current — that block is the single source of truth and gets corrected independ Both are read-only. Neither touches the `.mpr`. Both refuse to report a clean sweep from a stale or empty catalog rather than silently passing. -**Stale-catalog self-heal:** if `graph-sweep.sh` faults because `catalog_meta.build_mode` is not -`full`, you may run `./mxcli -p -c 'REFRESH CATALOG FULL'` yourself and retry once — this -rebuilds the catalog database, it does not write to `.mpr` content, so it's outside what the +**Stale-catalog self-heal:** if `graph-sweep.sh` faults because the catalog is stale, its refs table +is empty, or its refs carry no `RefKind` (its `build_mode` label is reported, not judged — mxcli v0.24+ writes +`source` for a complete catalog), you may run `./mxcli -p -c 'REFRESH CATALOG FULL'` +yourself and retry once — this rebuilds the catalog database, it does not write to `.mpr` content, so it's outside what the project's write-approval rule protects. If the retry still faults, report it; don't loop. ## Four hard rules — STOP, not suggestions diff --git a/project-bin/app-facts.sh b/project-bin/app-facts.sh index ac8b7eb1..534d3263 100755 --- a/project-bin/app-facts.sh +++ b/project-bin/app-facts.sh @@ -246,10 +246,13 @@ fi q() { sqlite3 -noheader "$DB" "$1"; } BUILD_MODE="$(q "SELECT Value FROM catalog_meta WHERE Key='build_mode';")" -if [ "$BUILD_MODE" != "full" ]; then - echo "FAULT: catalog build_mode is '$BUILD_MODE' after graph-report; refs/activities are not trustworthy." >&2 - exit 2 -fi +# The build_mode LABEL is not checked (fixed 2026-10-13, field report 2026-10-11): mxcli v0.24.0 +# labels a complete catalog 'source', and the old `!= full` test refused it on every run. What +# this script needs is checked instead, below: the tables by name, non-empty refs/activities, +# and at least one refs row that carries a RefKind (the dependency facts group by it). The label +# is recorded in manifest.json as found, so a reader can still see it. Zero 'call' edges is NOT a +# fault: an early module reached only from pages and buttons has none, and its call graph is +# legitimately empty — requiring them (the first cut of this fix) refused such apps. MISSING="" for t in modules_data entities_data microflows_data activities_data pages_data scheduled_events_data refs catalog_meta projects; do [ "$(q "SELECT count(*) FROM sqlite_master WHERE name='$t';")" = "1" ] || MISSING="$MISSING $t" @@ -264,10 +267,17 @@ N_MODULES=$(q "SELECT count(*) FROM modules_data;") N_REFS=$(q "SELECT count(*) FROM refs;") N_ACTS=$(q "SELECT count(*) FROM activities_data;") if [ "${N_MODULES:-0}" -eq 0 ] || [ "${N_REFS:-0}" -eq 0 ] || [ "${N_ACTS:-0}" -eq 0 ]; then - echo "FAULT: catalog is vacuous (modules=$N_MODULES refs=$N_REFS activities=$N_ACTS). An empty graph is not a clean app." >&2 + echo "FAULT: catalog is vacuous (modules=$N_MODULES refs=$N_REFS activities=$N_ACTS, build_mode '${BUILD_MODE:-unset}'). An empty graph is not a clean app." >&2 + exit 2 +fi +KINDS="$(q "SELECT k || '=' || n FROM (SELECT COALESCE(NULLIF(RefKind,''),'(none)') k, count(*) n FROM refs GROUP BY k ORDER BY n DESC LIMIT 8);" | paste -sd' ' -)" +N_KINDED=$(q "SELECT count(*) FROM refs WHERE COALESCE(RefKind,'') <> '';") +if [ "${N_KINDED:-0}" -eq 0 ]; then + echo "FAULT: catalog refs carry no RefKind (build_mode '${BUILD_MODE:-unset}', $N_REFS refs by kind: ${KINDS:-none})." >&2 + echo " The dependency and call-graph facts are read by kind. Refusing to write facts without one." >&2 exit 2 fi -log "catalog ok: modules=$N_MODULES refs=$N_REFS activities=$N_ACTS (${T1}s)" +log "catalog ok: build_mode=${BUILD_MODE:-unset} modules=$N_MODULES refs=$N_REFS (${KINDS:-none}) activities=$N_ACTS (${T1}s)" # --------------------------------------------------------------------------------------------- # Step 2 — inventory + dependencies, straight from the catalog through the REAL module column. @@ -546,7 +556,7 @@ fi FACTS_DIR="$FACTS_DIR" MX_VERSION="$MX_VERSION" MPR_BASE="$MPR_BASE" DB="$DB" \ T1="$T1" T2="$T2" T3="$T3" TALL="$(( $(date +%s) - START_ALL ))" \ LOOP_STATUS="$LOOP_STATUS" LOOP_REASON="$LOOP_REASON" N_CAND="$N_CAND" N_DESC="$N_DESC" N_FAILED="$N_FAILED" N_RULES="$N_RULES" \ -N_MODULES="$N_MODULES" N_REFS="$N_REFS" N_ACTS="$N_ACTS" WORKERS="$WORKERS" MAX_LOOPS="$MAX_LOOPS" "$PY" - <<'PY' +BUILD_MODE="$BUILD_MODE" N_MODULES="$N_MODULES" N_REFS="$N_REFS" N_ACTS="$N_ACTS" WORKERS="$WORKERS" MAX_LOOPS="$MAX_LOOPS" "$PY" - <<'PY' import json, os, datetime, hashlib F = os.environ["FACTS_DIR"] def sha(p): @@ -563,7 +573,7 @@ man = { "generated": datetime.datetime.now().astimezone().isoformat(timespec="seconds"), "mpr": os.environ["MPR_BASE"], "mpr_sha256_16": sha(mp) if os.path.exists(mp) else "", "mxcli": os.environ["MX_VERSION"], - "catalog": {"modules": int(os.environ["N_MODULES"]), "refs": int(os.environ["N_REFS"]), "activities": int(os.environ["N_ACTS"]), "build_mode": "full"}, + "catalog": {"modules": int(os.environ["N_MODULES"]), "refs": int(os.environ["N_REFS"]), "activities": int(os.environ["N_ACTS"]), "build_mode": os.environ.get("BUILD_MODE", "")}, "timings_s": {"catalog_and_graph_report": int(os.environ["T1"]), "inventory_and_dependencies": int(os.environ["T2"]), "loops": int(os.environ["T3"]), "total": int(os.environ["TALL"])}, "options": {"workers": int(os.environ["WORKERS"]), "max_loops": int(os.environ["MAX_LOOPS"])}, "sections": { diff --git a/project-bin/conformance-check.sh b/project-bin/conformance-check.sh index ebbb94a8..b76031cb 100755 --- a/project-bin/conformance-check.sh +++ b/project-bin/conformance-check.sh @@ -31,13 +31,22 @@ # Nothing here blocks that did not block before, and no exit code got more severe: a missing # ledger now exits 3 (NOT MEASURED) or 4 (NOT APPLICABLE) instead of 2 (FAULT). # -# Ledger path shapes, all read, first is canonical: -# architecture/modules//coverage-ledger.md canonical (per-module) -# architecture/modules/-coverage-ledger.md flat module layout -# architecture/coverage-ledger.md single-BRD project +# Ledger path shapes, all read: +# architecture/modules//coverage-ledger/*.md one ledger per BRD (coverage-check-all.sh's +# form); EVERY file is measured, one report +# per BRD, and the coverage-ledger.md beside +# the directory is treated as its index +# architecture/modules//coverage-ledger.md canonical (per-module) +# architecture/modules/-coverage-ledger.md flat module layout +# architecture/coverage-ledger.md single-BRD project # # Usage: # bin/conformance-check.sh [--module ] [--update-baseline] [--quiet] +# LEDGER_FILE= bin/conformance-check.sh [--module ] ... +# +# LEDGER_FILE overrides discovery: one ledger file, or a directory whose *.md files are all +# measured (one per BRD). Relative to the project root. A path that is neither is +# a FAULT, never a silent fall-back to another ledger. # # Exit: 0 clean or baseline written · 1 regression · 2 instrument fault (refuses to guess) # 3 NOT MEASURED (no ledger, but a spec exists) · 4 NOT APPLICABLE (no spec at all) @@ -60,7 +69,7 @@ while [ $# -gt 0 ]; do --module) MODULE="${2:-}"; shift 2 ;; --update-baseline) UPDATE_BASELINE=1; shift ;; --quiet) QUIET=1; shift ;; - -h|--help) sed -n '2,45p' "$0"; exit 0 ;; + -h|--help) sed -n '2,52p' "$0"; exit 0 ;; *) echo "unknown argument: $1" >&2; exit 2 ;; esac done @@ -80,20 +89,47 @@ mkdir -p "$OUTDIR" # LEDGER_COUNT is tracked separately rather than read back as ${#LEDGERS[@]}: under `set -u`, # bash 3.2 (stock macOS) treats an EMPTY array as unset and aborts on that expansion — which is # precisely the case this block exists to handle gracefully. +# +# PER-BRD DIRECTORIES (fixed 2026-10-13, field report 2026-10-11): a +# module that splits its ledger one-per-BRD keeps the rows in /coverage-ledger/.md +# and turns /coverage-ledger.md into an index. This script only ever read the index, so +# every such module ended in "FAULT: 0 measurable rows extracted from 1 ledger(s)." — while the +# operator's LEDGER_FILE workaround on ONE per-BRD file measured 128 rows (66 measured, 66 OK). +# coverage-preflight.sh already read the directory first (2026-09-26); this is the same rule: +# the directory wins over the index beside it, and every file in it is measured. LEDGERS=() LEDGER_COUNT=0 -if [ -n "$MODULE" ]; then - for c in "${LEDGER_FILE:-}" \ - "architecture/modules/$MODULE/coverage-ledger.md" \ - "architecture/modules/$MODULE-coverage-ledger.md" \ - "architecture/coverage-ledger.md"; do - [ -n "$c" ] && [ -f "$c" ] && { LEDGERS=("$c"); LEDGER_COUNT=1; break; } - done +add_ledger() { LEDGERS[$LEDGER_COUNT]="$1"; LEDGER_COUNT=$((LEDGER_COUNT+1)); } +has_md() { [ -d "$1" ] && ls "$1"/*.md >/dev/null 2>&1; } +add_dir() { local f; for f in "$1"/*.md; do [ -f "$f" ] && add_ledger "$f"; done; } +if [ -n "${LEDGER_FILE:-}" ]; then + if has_md "$LEDGER_FILE"; then add_dir "$LEDGER_FILE" + elif [ -f "$LEDGER_FILE" ]; then add_ledger "$LEDGER_FILE" + else + echo "FAULT: LEDGER_FILE=$LEDGER_FILE is neither a ledger file nor a directory of *.md ledgers" >&2 + echo " (resolved against $PROJECT_ROOT). Refusing to fall back to another ledger." >&2 + exit 2 + fi +elif [ -n "$MODULE" ]; then + if has_md "architecture/modules/$MODULE/coverage-ledger"; then + add_dir "architecture/modules/$MODULE/coverage-ledger" + else + for c in "architecture/modules/$MODULE/coverage-ledger.md" \ + "architecture/modules/$MODULE-coverage-ledger.md" \ + "architecture/coverage-ledger.md"; do + [ -f "$c" ] && { add_ledger "$c"; break; } + done + fi else + for d in architecture/modules/*/coverage-ledger; do + has_md "$d" && add_dir "$d" + done for c in architecture/modules/*/coverage-ledger.md \ architecture/modules/*-coverage-ledger.md \ architecture/coverage-ledger.md; do - if [ -f "$c" ]; then LEDGERS[$LEDGER_COUNT]="$c"; LEDGER_COUNT=$((LEDGER_COUNT+1)); fi + [ -f "$c" ] || continue + has_md "${c%.md}" && continue # the index of a per-BRD directory measured above + add_ledger "$c" done fi @@ -149,6 +185,8 @@ fi # SHOW/DESCRIBE in the acceptance cell are measurable; the rest are counted, not run. # The module label comes from the path, and the path has three shapes (above), so read it per # shape instead of assuming the parent directory is always the module. +# A per-BRD file /coverage-ledger/.md belongs to and carries a BRD label; +# every other shape has no BRD label ("-": read splits on tabs, so an empty field would shift). ledger_module() { local ledger="$1" base parent base="$(basename "$ledger")" @@ -157,26 +195,38 @@ ledger_module() { *-coverage-ledger.md) printf '%s\n' "${base%-coverage-ledger.md}"; return ;; esac case "$parent" in - architecture) printf '%s\n' "${MODULE:-project}"; return ;; - *) printf '%s\n' "$parent"; return ;; + architecture) printf '%s\n' "${MODULE:-project}"; return ;; + coverage-ledger) printf '%s\n' "$(basename "$(dirname "$(dirname "$ledger")")")"; return ;; + *) printf '%s\n' "$parent"; return ;; + esac +} +ledger_brd() { + case "$(basename "$(dirname "$1")")" in + coverage-ledger) printf '%s\n' "$(basename "$1" .md)" ;; + *) printf '%s\n' "-" ;; esac } +# Rows are keyed (module, pointer) in the baseline. A JSON pointer is relative to ITS BRD, so two +# per-BRD ledgers of one module both have `/microflows/0`; their pointer key carries the BRD id. extract() { - local ledger="$1" mod + local ledger="$1" mod brd mod="$(ledger_module "$ledger")" - awk -F'|' -v mod="$mod" ' + brd="$(ledger_brd "$ledger")" + awk -F'|' -v mod="$mod" -v brd="$brd" -v file="$ledger" ' NF==9 && $2 ~ /^ *`\// { total++ ptr=$2; acc=$7; st=$8 gsub(/^ +| +$/, "", ptr); gsub(/^ +| +$/, "", st) + if (brd != "-") ptr = brd ":" ptr if (match(acc, /`(SHOW|DESCRIBE)[^`]*`/)) { cmd = substr(acc, RSTART+1, RLENGTH-2) - print "ROW\t" mod "\t" ptr "\t" st "\t" cmd + print "ROW\t" mod "\t" brd "\t" ptr "\t" st "\t" cmd } else { - print "PROSE\t" mod "\t" ptr "\t" st "\t" + print "PROSE\t" mod "\t" brd "\t" ptr "\t" st "\t" } } + END { if (!total) print "NOTABLE\t" file } ' "$ledger" } @@ -184,10 +234,39 @@ ROWS="$(for l in "${LEDGERS[@]}"; do [ -f "$l" ] && extract "$l"; done)" MEASURABLE=$(printf '%s\n' "$ROWS" | grep -c '^ROW' || true) PROSE=$(printf '%s\n' "$ROWS" | grep -c '^PROSE' || true) +NOTABLE="$(printf '%s\n' "$ROWS" | awk -F'\t' '$1=="NOTABLE"{print $2}')" + +# A ledger file with no ledger table is, in practice, an index: the coverage-ledger.md a module +# keeps beside its per-BRD directory, or one that points at another module's per-BRD file. +# Say where its rows are rather than "0 measurable rows", which reads as a broken instrument. +explain_index() { + local f="$1" dir links + dir="${f%.md}" + echo " $f has no ledger table rows — it is an index, not a ledger." >&2 + if has_md "$dir"; then + echo " Its rows are in $dir/:" >&2 + for l in "$dir"/*.md; do echo " $l" >&2; done + echo " Unset LEDGER_FILE (the directory is found by itself) or set LEDGER_FILE=$dir" >&2 + return + fi + links="$(grep -oE '\]\([^)#]+\.md\)' "$f" 2>/dev/null | sed -E 's/^\]\(//; s/\)$//' | sort -u)" + if [ -n "$links" ]; then + echo " It links to these ledgers (relative to $(dirname "$f")/) — rows are in them:" >&2 + printf '%s\n' "$links" | sed 's/^/ /' >&2 + echo " Measure them with LEDGER_FILE=." >&2 + fi +} # Non-vacuous guard: an instrument that measured nothing must not report "all clean". if [ "$MEASURABLE" -eq 0 ]; then - echo "FAULT: 0 measurable rows extracted from ${#LEDGERS[@]} ledger(s)." >&2 + NNOTABLE=$(printf '%s' "$NOTABLE" | grep -c . || true) + if [ "$NNOTABLE" -gt 0 ] && [ "$NNOTABLE" -eq "$LEDGER_COUNT" ]; then + echo "FAULT: nothing to measure — the ledger read is an index, not a ledger table." >&2 + while IFS= read -r f; do [ -n "$f" ] && explain_index "$f"; done <<< "$NOTABLE" + echo " Refusing to report a clean run over zero measurements." >&2 + exit 2 + fi + echo "FAULT: 0 measurable rows extracted from $LEDGER_COUNT ledger(s)." >&2 echo " Either the ledger table shape changed or the glob matched nothing." >&2 echo " Refusing to report a clean run over zero measurements." >&2 exit 2 @@ -281,7 +360,7 @@ probe() { RESULTS="" n=0 -while IFS=$'\t' read -r kind mod ptr status cmd; do +while IFS=$'\t' read -r kind mod brd ptr status cmd; do [ "$kind" = "ROW" ] || continue n=$((n+1)) [ "$QUIET" -eq 1 ] || printf '\r measuring %d/%d ' "$n" "$MEASURABLE" >&2 @@ -297,7 +376,7 @@ while IFS=$'\t' read -r kind mod ptr status cmd; do *) verdict="UNKNOWN-STATUS" ;; esac ;; esac - RESULTS+="$mod $ptr $status $observed $verdict $cmd"$'\n' + RESULTS+="$mod $ptr $status $observed $verdict $cmd $brd"$'\n' done 3<<< "$ROWS" <&3 [ "$QUIET" -eq 1 ] || printf '\r%*s\r' 30 '' >&2 @@ -346,8 +425,18 @@ NSEED=$(printf '%s' "$SEEDED" | grep -c . || true) # --- report ------------------------------------------------------------------------------ STAMP="$(date +%Y-%m-%d)" -REPORT="$OUTDIR/report-$STAMP.tsv" -{ printf 'module\tpointer\tstored\tobserved\tverdict\tcommand\n'; printf '%s\n' "$RESULTS"; } > "$REPORT" +# One report per BRD for per-BRD ledgers (report--.tsv), so measuring several BRDs +# does not leave only the last one on disk; rows from every other shape keep report-.tsv. +REPORTS="" +# while-read, not `for b in $(…)`: a BRD file name with a space must stay one BRD. +while IFS= read -r b; do + [ -n "$b" ] || continue + if [ "$b" = "-" ]; then REPORT="$OUTDIR/report-$STAMP.tsv"; else REPORT="$OUTDIR/report-$STAMP-$b.tsv"; fi + { printf 'module\tpointer\tstored\tobserved\tverdict\tcommand\n' + printf '%s\n' "$RESULTS" | awk -F'\t' -v b="$b" 'BEGIN{OFS="\t"} $7==b {print $1,$2,$3,$4,$5,$6}' + } > "$REPORT" + REPORTS+="$REPORT"$'\n' +done <<< "$(printf '%s\n' "$RESULTS" | awk -F'\t' 'NF{print $7}' | sort -u)" count() { printf '%s\n' "$RESULTS" | awk -F'\t' -v v="$1" '$5==v' | grep -c . || true; } OK=$(count OK); STALE=$(count STALE); UNDER=$(count UNDERSTATED); UNK=$(count UNKNOWN-STATUS) @@ -363,7 +452,10 @@ echo " UNDERSTATED $UNDER" [ "$UNRUN" -gt 0 ] && echo " UNRUNNABLE $UNRUN (backticked prose, or a probe mxcli rejects — fix the probe; see the report)" [ "$TMO" -gt 0 ] && echo " TIMEOUT $TMO (>${TIMEOUT_S}s — never counted as a pass)" [ "$UNK" -gt 0 ] && echo " UNKNOWN-STATUS $UNK" -echo " report $REPORT" +while IFS= read -r r; do [ -n "$r" ] && echo " report $r"; done <<< "$REPORTS" +if [ -n "$NOTABLE" ]; then + while IFS= read -r f; do [ -n "$f" ] && echo " no table (index?) $f"; done <<< "$NOTABLE" +fi [ "$NSEED" -gt 0 ] && echo " newly baselined $NSEED (no baseline row yet — regressions measured from the next run)" if [ "$STALE" -gt 0 ] || [ "$UNDER" -gt 0 ]; then diff --git a/project-bin/graph-sweep.sh b/project-bin/graph-sweep.sh index 41f1b65c..35e4647e 100755 --- a/project-bin/graph-sweep.sh +++ b/project-bin/graph-sweep.sh @@ -26,7 +26,7 @@ # is an intent question, answered against the module brief, not here. # # Usage: bin/graph-sweep.sh [--module ] [--min-elements N] [--tsv] -# Exit: 0 swept · 2 instrument fault (stale catalog, empty graph, missing db) +# Exit: 0 swept · 2 instrument fault (stale catalog, empty or untyped graph, missing db) set -uo pipefail @@ -110,20 +110,41 @@ if [ "$CAT_MTIME" != "$MPR_MTIME" ]; then exit 2 fi -if [ "$BUILD_MODE" != "full" ]; then - echo "FAULT: catalog build_mode is '$BUILD_MODE'; reference edges require 'full'." >&2 - echo " Rebuild: ./mxcli -p $MPR -c 'REFRESH CATALOG FULL'" >&2 - exit 2 -fi - -# --- guard 2: non-vacuous ------------------------------------------------------------------ +# --- guard 2: non-vacuous, measured on the edges this sweep reads ------------------------- # An empty graph produces zero findings, which is indistinguishable from a clean model. +# +# WHY NOT `build_mode` (fixed 2026-10-13, reported from the field 2026-10-11): this guard used +# to refuse any catalog whose catalog_meta.build_mode was not the literal 'full'. mxcli v0.24.0 +# writes 'source' even after `REFRESH CATALOG FULL`, and that catalog held 5,368 typed refs +# rows. So the sweep FAULTed on every run on that mxcli, on a fresh, complete catalog: +# FAULT: catalog build_mode is 'source'; reference edges require 'full'. +# The label was a proxy for "the reference edges are there". Test that directly instead: the +# wiring finding reads every refs row, grouped by module, and the orphan finding reads the +# inbound kinds in $READ_KINDS (shared with the ORPHANS query below). A catalog built without +# edges (the old 'fast' mode) has an empty refs table; one whose rows carry no RefKind cannot +# answer the orphan question either. Those two FAULT, naming the mode and the per-kind counts. +# +# Requiring 'call' specifically (the first cut of this fix) was a regression: an early greenfield +# module reached only from buttons and pages has action/show_page edges and no microflow-to- +# microflow call at all, and master swept it fine. So any kinded edge is enough to run; a catalog +# with none of the orphan-read kinds still sweeps, with a warning that its orphan list is moot. +READ_KINDS="'call','action','menu_item','show_page','datasource'" REFS=$(q "SELECT COUNT(*) FROM refs;") OBJS=$(q "SELECT COUNT(*) FROM objects;") +KINDS="$(q "SELECT k || '=' || n FROM (SELECT COALESCE(NULLIF(RefKind,''),'(none)') k, count(*) n FROM refs GROUP BY k ORDER BY n DESC LIMIT 8);" | paste -sd' ' -)" if [ "${REFS:-0}" -eq 0 ] || [ "${OBJS:-0}" -eq 0 ]; then - echo "FAULT: graph is empty (refs=$REFS objects=$OBJS). Refusing to report a clean sweep." >&2 + echo "FAULT: graph is empty (refs=$REFS objects=$OBJS, build_mode '${BUILD_MODE:-unset}'). Refusing to report a clean sweep." >&2 + echo " Rebuild: ./mxcli -p $MPR -c 'REFRESH CATALOG FULL'" >&2 + exit 2 +fi +KINDED=$(q "SELECT COUNT(*) FROM refs WHERE COALESCE(RefKind,'') <> '';") +if [ "${KINDED:-0}" -eq 0 ]; then + echo "FAULT: catalog refs carry no RefKind (build_mode '${BUILD_MODE:-unset}', $REFS refs by kind: ${KINDS:-none})." >&2 + echo " The orphan finding is read by kind. Refusing to report a sweep from untyped edges." >&2 + echo " Rebuild: ./mxcli -p $MPR -c 'REFRESH CATALOG FULL'" >&2 exit 2 fi +READ_EDGES=$(q "SELECT COUNT(*) FROM refs WHERE RefKind IN ($READ_KINDS);") WIRING_FILTER="" ORPHAN_FILTER="" @@ -179,7 +200,7 @@ WHERE m.ModuleName NOT IN ( AND NOT EXISTS ( SELECT 1 FROM refs r WHERE r.TargetName = m.QualifiedName - AND r.RefKind IN ('call','action','menu_item','show_page','datasource')) + AND r.RefKind IN ($READ_KINDS)) $ORPHAN_FILTER ORDER BY m.ModuleName, m.Name; ")" @@ -195,7 +216,7 @@ fi N_UNWIRED=$(printf '%s\n' "$WIRING" | awk -F'|' 'NF && $3==0' | grep -c . || true) N_ORPH=$(printf '%s\n' "$ORPHANS" | grep -c . || true) -echo "graph-sweep · catalog $CAT_MTIME · $OBJS objects · $REFS typed edges" +echo "graph-sweep · catalog $CAT_MTIME (build_mode ${BUILD_MODE:-unset}) · $OBJS objects · $REFS typed edges (${KINDS:-none})" echo echo " module wiring (>= $MIN_ELEMENTS elements)" printf ' %-24s %8s %9s %10s\n' MODULE ELEMENTS INBOUND OUTBOUND @@ -203,6 +224,7 @@ printf '%s\n' "$WIRING" | awk -F'|' 'NF {printf " %-24s %8s %9s %10s%s\n", $1 echo echo " orphaned microflows (no call / action / nav inbound, entry-point prefixes excluded): $N_ORPH" +[ "${READ_EDGES:-0}" -eq 0 ] && echo " WARNING: the catalog has no call/action/menu_item/show_page/datasource edges, so every microflow reads as orphaned here; check the edge kinds above." if [ "$N_ORPH" -gt 0 ]; then printf '%s\n' "$ORPHANS" | awk -F'|' 'NF {printf " %-56s %s activities\n", $1, $2}' | head -40 [ "$N_ORPH" -gt 40 ] && echo " ... and $((N_ORPH-40)) more" diff --git a/project-bin/review-module.sh b/project-bin/review-module.sh index 87a6ade1..c9923dcb 100755 --- a/project-bin/review-module.sh +++ b/project-bin/review-module.sh @@ -302,40 +302,65 @@ if [ "$MODULE_KNOWN" -eq 0 ]; then fi # ── 1. Conformance: the ledger's claimed status vs the live model ─────────── +# A module whose ledger is split one-per-BRD gets one conformance report PER BRD +# (report--.tsv, conformance-check.sh since 2026-10-13), not one report-.tsv. +# Reading a single file here took one BRD's rows (newest-by-mtime) or, on --reuse, none at all +# — a per-BRD module would FAULT rung 13 of verify-module with "does not exist". So both paths +# below read every report of the run and slice them together. +# conf_slice OUT FILE... — header once, then this module's rows from every file. +conf_slice() { + local out="$1"; shift + awk -F'\t' -v m="$MODULE" 'NR==1 {print; next} FNR==1 {next} $1==m' "$@" > "$out" +} # The ledger is a claim. This measures it. A module with no ledger has made no claims to # measure, which is a FAULT and emphatically not a pass — "no ledger" is exactly the hole # that let projects ship with no coverage-ledger.md at all and read as silence. LEDGER="$ROOT/architecture/modules/$MODULE/coverage-ledger.md" -if [ ! -f "$LEDGER" ]; then +# A module may keep ONLY the per-BRD directory (/coverage-ledger/.md) with no index +# file beside it; that is a ledger, and conformance-check.sh / coverage-preflight.sh both read it. +LEDGER_PB_DIR="${LEDGER%.md}" +has_ledger() { [ -f "$LEDGER" ] || ls "$LEDGER_PB_DIR"/*.md >/dev/null 2>&1; } +if ! has_ledger; then fault_instrument "conformance (ledger claims vs live model)" conformance \ - "no architecture/modules/$MODULE/coverage-ledger.md" \ + "no architecture/modules/$MODULE/coverage-ledger.md or coverage-ledger/*.md" \ "This module has made no recorded claims, so none could be measured." \ "Its conformance is UNMEASURED, not clean: there is no row here saying" \ "'the model matches the decision' because there is no decision on file." elif [ -n "$REUSE_CONF" ]; then printf '\n\033[1m── conformance (reusing the caller'"'"'s run)\033[0m\n' case "$REUSE_CONF" in /*) RC_PATH="$REUSE_CONF" ;; *) RC_PATH="$ROOT/$REUSE_CONF" ;; esac - if [ ! -f "$RC_PATH" ]; then + # The reused path names the run's base report (report-.tsv); its per-BRD siblings + # (report--.tsv) belong to the same run and are read with it. + # Counted, not read back as ${#RC_FILES[@]}: bash 3.2 (stock macOS) aborts on an empty + # array expansion under `set -u`. + RC_FILES=(); RC_N=0 + [ -f "$RC_PATH" ] && { RC_FILES[$RC_N]="$RC_PATH"; RC_N=$((RC_N+1)); } + for f in "${RC_PATH%.tsv}"-*.tsv; do [ -f "$f" ] && { RC_FILES[$RC_N]="$f"; RC_N=$((RC_N+1)); }; done + # The files actually read, project-relative, for the log fields — on a per-BRD run the base + # name passed in does not exist, so naming it would point a reader at nothing. + RC_LIST="" + for f in ${RC_FILES[@]+"${RC_FILES[@]}"}; do RC_LIST="${RC_LIST:+$RC_LIST, }${f#$ROOT/}"; done + if [ "$RC_N" -eq 0 ]; then c_warn " ! INSTRUMENT FAULT"; echo " — $REUSE_CONF does not exist" echo " The caller said it had already measured conformance and it had not." printf 'conformance (reused)\tFAULT\t2\t-\t(%s missing)\n' "$REUSE_CONF" >> "$SUMMARY" note_instrument conformance fault 2 "" "" "reuse requested but $REUSE_CONF does not exist" FAULTED=$((FAULTED+1)) else - awk -F'\t' -v m="$MODULE" 'NR==1 || $1==m' "$RC_PATH" > "$OUTDIR/conformance.tsv" - RC_ROWS=$(awk -F'\t' -v m="$MODULE" '$1==m' "$RC_PATH" | grep -c . || true) + conf_slice "$OUTDIR/conformance.tsv" "${RC_FILES[@]}" + RC_ROWS=$(awk -F'\t' -v m="$MODULE" 'FNR>1 && $1==m' "${RC_FILES[@]}" | grep -c . || true) if [ "${RC_ROWS:-0}" -eq 0 ]; then rm -f "$OUTDIR/conformance.tsv" c_warn " ! INSTRUMENT FAULT"; echo " — $REUSE_CONF carries no rows for $MODULE" echo " Reusing it would report zero measurements as a clean module." printf 'conformance (reused)\tFAULT\t2\t-\t(no rows for %s)\n' "$MODULE" >> "$SUMMARY" - note_instrument conformance fault 2 "" "${REUSE_CONF}" \ + note_instrument conformance fault 2 "" "$RC_LIST" \ "reused report carries no rows for $MODULE — refusing to report a clean run over zero measurements" FAULTED=$((FAULTED+1)) else - c_ok " ✓ reused"; echo " ($RC_ROWS rows from ${REUSE_CONF})" - printf 'conformance (reused)\tPASS\t0\t-\t%s\n' "$REUSE_CONF" >> "$SUMMARY" - note_instrument conformance pass 0 "" "$REUSE_CONF" "" + c_ok " ✓ reused"; echo " ($RC_ROWS rows from $RC_N report(s): $RC_LIST)" + printf 'conformance (reused)\tPASS\t0\t-\t%s\n' "$RC_LIST" >> "$SUMMARY" + note_instrument conformance pass 0 "" "$RC_LIST" "" fi fi else @@ -359,13 +384,18 @@ else echo " conformance did not complete — NOT slicing an older report in as its result." else CONF_TSV="$(ls -t "$ROOT"/docs/conformance/report-*.tsv 2>/dev/null | head -1)" + CONF_NEW=(); CONF_N=0 + for f in "$ROOT"/docs/conformance/report-*.tsv; do + [ -f "$f" ] && [ "$f" -nt "$OUTDIR/.started" ] && { CONF_NEW[$CONF_N]="$f"; CONF_N=$((CONF_N+1)); } + done if [ -z "$CONF_TSV" ]; then echo " no docs/conformance/report-*.tsv to slice; the report will carry a fault row." - elif [ ! "$CONF_TSV" -nt "$OUTDIR/.started" ]; then + elif [ "$CONF_N" -eq 0 ]; then echo " newest conformance report predates this run — NOT slicing it in." echo " (${CONF_TSV#$ROOT/} is older than $STAMP; it measured a different run.)" else - awk -F'\t' -v m="$MODULE" 'NR==1 || $1==m' "$CONF_TSV" > "$OUTDIR/conformance.tsv" + # Every report this run wrote: one, or one per BRD for a per-BRD ledger. + conf_slice "$OUTDIR/conformance.tsv" "${CONF_NEW[@]}" fi fi fi @@ -389,7 +419,7 @@ cand="$(_tool coverage-check.sh)" # The BRD is named by the ledger itself; guessing with `ls | head -1` picks another # module's BRD and reports its leaves as this module's coverage. BRD="" -if [ -f "$LEDGER" ]; then +if has_ledger; then # ALL the BRDs the ledger names, not just the first — and the COUNT is reported, because # coverage-check.sh measures one BRD per run and a module with five of them was silently # having one fifth of its requirements measured and printed as the module's coverage @@ -404,7 +434,7 @@ if [ -f "$LEDGER" ]; then while IFS= read -r cand; do [ -n "$cand" ] && [ -f "$ROOT/$cand" ] && BRD_ALL="$BRD_ALL$cand"$'\n' done </dev/null | sort -u) +$(cat "$LEDGER" "$LEDGER_PB_DIR"/*.md 2>/dev/null | grep -oE '[A-Za-z0-9_/.-]*\.brd\.json' | sort -u) EOF BRD_N=$(printf '%s' "$BRD_ALL" | grep -c . || true) BRD_REL="$(printf '%s' "$BRD_ALL" | head -1)" @@ -414,9 +444,9 @@ if [ -z "$COV" ]; then fault_instrument "coverage (BRD leaves: UNCLAIMED/PHANTOM/DOUBLE)" coverage \ "coverage-check.sh not reachable (neither bin/ nor the toolkit)" \ "Requirement traceability for $MODULE is UNMEASURED, not clean." -elif [ ! -f "$LEDGER" ]; then +elif ! has_ledger; then fault_instrument "coverage (BRD leaves: UNCLAIMED/PHANTOM/DOUBLE)" coverage \ - "no coverage-ledger.md for $MODULE" \ + "no coverage-ledger.md or coverage-ledger/*.md for $MODULE" \ "Requirement traceability for $MODULE is UNMEASURED, not clean." elif [ -z "$BRD" ]; then fault_instrument "coverage (BRD leaves: UNCLAIMED/PHANTOM/DOUBLE)" coverage \ diff --git a/project-bin/verify-module.sh b/project-bin/verify-module.sh index c5dde886..fc4681aa 100755 --- a/project-bin/verify-module.sh +++ b/project-bin/verify-module.sh @@ -215,8 +215,15 @@ _report_result() { # into coherence-cadence.sh and build-plan-status.sh. Observed on a real field run: # `mode=static-only pages=0 fault=2` reported as PASS. Fault is checked FIRST because "did not # run" must never be readable as "ran and was clean". + # + # `stack:4` is test-stack-up.sh's "instrument blocked": the port `mxcli run --local` needs is + # held by another process. Its own header says to record that as blocked, not as a test + # failure — the app was never measured, so it is a FAULT, not a FINDING. Scoped to the stack + # rung's kind on purpose: exit 4 means something else elsewhere (conformance-check.sh: NOT + # APPLICABLE), so it is not a global remap. case "$kind:$rc" in *:2) verdict=FAULT ;; + stack:4) verdict=FAULT ;; info:*) verdict=INFO ;; *:0) verdict=PASS ;; *) verdict=FINDING ;; @@ -235,6 +242,7 @@ _report_result() { # run -- # kind=gate exit 1 => finding, exit 2 => instrument fault # kind=info never fails the pass; recorded for the trend line +# kind=stack as gate, plus exit 4 (test-stack-up: port held by another process) => fault # Blocks until the instrument finishes. run() { local name="$1" log="$2" kind="$3" tmo="$4"; shift 5 # the literal -- @@ -376,7 +384,7 @@ echo "══ verify-module: $MODULE ══ $STAMP" # so the runtime instruments below cannot be pointed at a guessed one. STACK_OK=0 if [ -x "$(_tool test-stack-up.sh)" ]; then - run "stack (test-stack-up --check)" "$OUTDIR/00-stack.log" gate 60 -- \ + run "stack (test-stack-up --check)" "$OUTDIR/00-stack.log" stack 60 -- \ "$(_tool test-stack-up.sh)" --check grep -q "^stack (test-stack-up --check) PASS" "$SUMMARY" && STACK_OK=1 # "app up with a caveat" is not "app down", and reporting it as one sends the reader @@ -444,8 +452,16 @@ fi # requirements to check against" is precisely the hole that let several projects ship with no # coverage ledger at all, so coverage-check.sh and the review agent had nothing to run and their # absence read as silence. -LEDGER="${LEDGER_FILE:-$ROOT/architecture/modules/$MODULE/coverage-ledger.md}" -[ -f "$LEDGER" ] || LEDGER="$ROOT/architecture/coverage-ledger.md" +# A ledger is a file OR a per-BRD directory (/coverage-ledger/.md) — LEDGER_FILE +# documents both (conformance-check.sh, coverage-preflight.sh). Testing only `-f` sent a directory +# LEDGER_FILE silently to architecture/coverage-ledger.md (review of #307, 2026-10-13). +resolve_ledger_hint() { + local l="${LEDGER_FILE:-$ROOT/architecture/modules/$MODULE/coverage-ledger.md}" + if [ -z "${LEDGER_FILE:-}" ] && [ ! -f "$l" ] && [ -d "${l%.md}" ]; then l="${l%.md}"; fi + [ -f "$l" ] || [ -d "$l" ] || l="$ROOT/architecture/coverage-ledger.md" + printf '%s\n' "$l" +} +LEDGER="$(resolve_ledger_hint)" BRD="${BRD_FILE:-}" if [ -z "$BRD" ]; then for pat in "$ROOT"/analysis/*/brd/*.brd.json "$ROOT"/analysis/*/knowledge-base/brd/*.brd.json; do @@ -510,6 +526,9 @@ run_join_all # a second time in the same pass buys nothing. The path is passed unconditionally: if rung 1 faulted # the file is absent, review says so and FAULTS, which is the truth — nothing was measured — and it # costs seconds instead of minutes. +# The path names the run's BASE report; a per-BRD ledger writes report--.tsv instead +# (and no base file), and review-module.sh reads every report-*.tsv sibling with it — so +# the reuse finds a per-BRD module's rows rather than FAULTing on "does not exist" (2026-10-13). if [ -x "$(_tool review-module.sh)" ]; then run "review (model vs decided — report.json)" "$OUTDIR/13-review.log" info 600 -- \ "$(_tool review-module.sh)" "$MODULE" --json-only \ diff --git a/skills/coverage-ledger.md b/skills/coverage-ledger.md index 431c68ea..88674a5a 100644 --- a/skills/coverage-ledger.md +++ b/skills/coverage-ledger.md @@ -14,11 +14,12 @@ ## Where the ledger lives -Three path shapes exist in real projects. All three are read by `project-bin/conformance-check.sh` -and `project-bin/coverage-preflight.sh`; the first is canonical. +Four path shapes exist in real projects. All four are read by `project-bin/conformance-check.sh` +and `project-bin/coverage-preflight.sh`; the per-module file is canonical. | Path | When | Status | |---|---|---| +| `architecture/modules//coverage-ledger/.md` | module with several BRDs, split one ledger per BRD (`coverage-check-all.sh`'s form) | accepted — wins over the `coverage-ledger.md` beside it, which becomes the index | | `architecture/modules//coverage-ledger.md` | multi-module project | **canonical** | | `architecture/modules/-coverage-ledger.md` | project whose `architecture/modules/` is flat | accepted | | `architecture/coverage-ledger.md` | single-BRD project | accepted | @@ -30,9 +31,20 @@ together), and changing canon would invalidate every ledger already written to i The flat shape is read because it is what projects actually have: PROJECT-A's `architecture/modules/` contains `Approval-brief.md`, `ProductNumbers.md`, `Common-brief.md` and no subdirectories at all. An instrument that globs only the canonical shape tells such a project it has no ledger even when -it has one. **When no ledger is found, the instruments print all three paths they tried** — a +it has one. **When no ledger is found, the instruments print all the paths they tried** — a "not found" that does not name what it looked for is not actionable. +The per-BRD directory is read because a module with several BRDs splits its ledger that way and +keeps `coverage-ledger.md` as an index. Conformance measures every `.md` in it and writes +one `docs/conformance/report--.tsv` per BRD; baseline pointers carry the BRD +(`:`), because the same JSON pointer recurs across BRDs. An index read on its own +(`LEDGER_FILE=…/coverage-ledger.md`, or a module whose index links to another module's per-BRD +file) is reported as an index, naming the files its rows are in. + +Bad, before the per-BRD read (field report, 2026-10-11) — every module with an index printed +`FAULT: 0 measurable rows extracted from 1 ledger(s).` while its rows sat one directory down. +`LEDGER_FILE=` overrides discovery in both instruments. + --- ## The four fallback levels — what an instrument does when there is no ledger diff --git a/tests/wave2/test-catalog-edge-guard.sh b/tests/wave2/test-catalog-edge-guard.sh new file mode 100755 index 00000000..0c276b13 --- /dev/null +++ b/tests/wave2/test-catalog-edge-guard.sh @@ -0,0 +1,214 @@ +#!/usr/bin/env bash +# Usage: bash test-catalog-edge-guard.sh [path-to-graph-sweep.sh] +# +# Fixture for the catalog guard in project-bin/graph-sweep.sh and project-bin/app-facts.sh +# (app-facts.sh is taken from beside the subject). +# +# The defect (issue #259, field report 2026-10-11, mxcli v0.24.0, Mendix 11.14.0): both scripts +# refused any catalog whose catalog_meta.build_mode was not the literal 'full'. mxcli v0.24.0 +# writes 'source' even after `REFRESH CATALOG FULL`, on a catalog holding 5,368 typed refs rows, +# so every sweep on that mxcli ended in (verbatim from the report — the golden input): +# FAULT: catalog build_mode is 'source'; reference edges require 'full'. +# The fix tests what the scripts read instead of the label: a non-empty refs table whose rows +# carry a RefKind. It FAULTs only on an empty or kind-less refs table. +# +# The first cut of the fix required a 'call' edge, which refused a model with no +# microflow-to-microflow calls at all (an early module reached only from pages and buttons) — +# a regression against master, caught in review. Case 2 guards it. +# +# Cases, each on BOTH layouts (single-tree, and two-tree with the .mpr and catalog under app/): +# 1. build_mode 'source' + call edges -> sweep runs, orphan finding still correct +# 2. build_mode 'full' + action edges only, zero calls -> sweep runs (as on master) +# 3. build_mode 'full' + kind-less refs -> FAULT naming the mode and the per-kind counts +# 4. build_mode 'fast' + empty refs -> FAULT, as before +# 4b. refs typed, but none of the orphan-read kinds -> sweeps, with the orphan-list WARNING +# 5. app-facts: 'source' + calls and action-only both pass the guard; kind-less FAULTs +# +# The catalog databases are SYNTHETIC: no real catalog.db is captured in this repo, and the +# report quotes only the FAULT line and the counts. Tables and columns are exactly the ones the +# two scripts query (catalog_meta key/value, refs SourceName/TargetName/RefKind, objects, +# microflows_data, and app-facts' nine required tables); RefKind values 'call' and 'action' are +# the ones the scripts' own queries name. The mxcli used by app-facts is a stub that copies a +# prepared catalog into place — no real mxcli runs. + +set -uo pipefail + +SUT="${1:-}" +[ -z "$SUT" ] && SUT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/project-bin/graph-sweep.sh" +if [ ! -f "$SUT" ]; then + echo "SKIP: subject not found at $SUT" + echo "SCORE: 0/0 — nothing to test" + exit 0 +fi +SUT="$(cd "$(dirname "$SUT")" && pwd)/$(basename "$SUT")" +AF="$(dirname "$SUT")/app-facts.sh" +command -v sqlite3 >/dev/null || { echo "SKIP: sqlite3 not on PATH"; echo "SCORE: 0/0"; exit 0; } + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok — $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL — $1"; [ -n "${2:-}" ] && echo " got: $2"; } + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/catalog-calls.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +# mkdb +mkdb() { + local db="$1" mode="$2" shape="$3" + mkdir -p "$(dirname "$db")"; rm -f "$db" + { + echo "CREATE TABLE catalog_meta (key TEXT, value TEXT);" + echo "INSERT INTO catalog_meta VALUES ('mpr_mod_time','2026-10-10T12:00:00.000');" + echo "INSERT INTO catalog_meta VALUES ('build_mode','$mode');" + echo "CREATE TABLE refs (SourceType TEXT, SourceName TEXT, TargetName TEXT, RefKind TEXT, ModuleName TEXT);" + echo "CREATE TABLE objects (QualifiedName TEXT, ModuleName TEXT, ObjectType TEXT);" + echo "CREATE TABLE microflows_data (QualifiedName TEXT, ModuleName TEXT, Name TEXT, ActivityCount INTEGER, MicroflowType TEXT);" + echo "CREATE TABLE modules_data (Name TEXT, Source TEXT, AppStoreVersion TEXT, AppStoreGuid TEXT);" + echo "CREATE TABLE entities_data (QualifiedName TEXT, ModuleName TEXT, EntityType TEXT);" + echo "CREATE TABLE activities_data (MicroflowQualifiedName TEXT, ModuleName TEXT, ActivityType TEXT);" + echo "CREATE TABLE pages_data (QualifiedName TEXT, ModuleName TEXT);" + echo "CREATE TABLE scheduled_events_data (QualifiedName TEXT, ModuleName TEXT);" + echo "CREATE TABLE projects (Name TEXT);" + echo "INSERT INTO modules_data VALUES ('Orders','', '', ''),('Billing','', '', '');" + echo "INSERT INTO activities_data VALUES ('Orders.SUB_Total','Orders','ActionActivity');" + for m in Orders.ACT_Save Orders.SUB_Total Orders.SUB_Unused Billing.ACT_Send; do + echo "INSERT INTO objects VALUES ('$m','${m%%.*}','MICROFLOW');" + echo "INSERT INTO microflows_data VALUES ('$m','${m%%.*}','${m#*.}',3,'MICROFLOW');" + done + case "$shape" in + calls|actions) + echo "INSERT INTO refs VALUES ('PAGE','Orders.Order_Edit','Orders.ACT_Save','action','Orders');" + echo "INSERT INTO refs VALUES ('MICROFLOW','Billing.ACT_Send','Orders.ACT_Save','action','Billing');" + [ "$shape" = calls ] && echo "INSERT INTO refs VALUES ('MICROFLOW','Orders.ACT_Save','Orders.SUB_Total','call','Orders');" ;; + otherkind) + echo "INSERT INTO refs VALUES ('MICROFLOW','Orders.ACT_Save','Orders.Order','retrieve','Orders');" ;; + kindless) + echo "INSERT INTO refs VALUES ('PAGE','Orders.Order_Edit','Orders.ACT_Save','','Orders');" + echo "INSERT INTO refs VALUES ('MICROFLOW','Orders.ACT_Save','Orders.SUB_Total',NULL,'Orders');" ;; + esac + } | sqlite3 "$db" +} + +# proj -> prints project dir; .mpr mtime pinned to match catalog_meta +proj() { + local p="$WORK/$1-$RANDOM" mdir + mkdir -p "$p" + if [ "$1" = two ]; then mdir="$p/app"; else mdir="$p"; fi + mkdir -p "$mdir" + : > "$mdir/Shop.mpr" + touch -t 202610101200.00 "$mdir/Shop.mpr" + printf '%s\n' "$p" +} +mdir() { if [ -d "$1/app" ]; then printf '%s/app\n' "$1"; else printf '%s\n' "$1"; fi; } +sweep() { ( cd "$1" && PROJECT_ROOT="$1" bash "$SUT" --min-elements 1 ) > "$WORK/out" 2>&1; echo $?; } +out() { tr '\n' ' ' < "$WORK/out" | cut -c1-400; } + +for layout in single two; do + echo "== graph-sweep · $layout-tree layout" + + # --- 1. 'source' with call edges: the field case -------------------------------------------- + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" source calls + rc="$(sweep "$P")" + if grep -q "FAULT: catalog build_mode is 'source'; reference edges require 'full'." "$WORK/out"; then + bad "[$layout] build_mode 'source' refused by label (issue #259 golden FAULT line)" "rc=$rc $(out)" + elif [ "$rc" = 0 ] && grep -q "graph-sweep ·" "$WORK/out"; then + ok "[$layout] build_mode 'source' with call edges is swept (rc 0)" + else + bad "[$layout] 'source' catalog with call edges did not sweep" "rc=$rc $(out)" + fi + if grep -q "Orders.SUB_Unused" "$WORK/out" && ! grep -q "Orders.SUB_Total " "$WORK/out"; then + ok "[$layout] orphan finding still reads call edges (SUB_Unused listed, called SUB_Total not)" + else + bad "[$layout] orphan finding wrong on a 'source' catalog" "$(out)" + fi + + # --- 2. no microflow-to-microflow calls at all: still sweeps (the review-caught regression) ---- + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" full actions + rc="$(sweep "$P")" + if [ "$rc" = 0 ] && grep -q "graph-sweep ·.*action=2" "$WORK/out" && grep -q "Orders.SUB_Total" "$WORK/out"; then + ok "[$layout] action-only catalog (zero calls) is swept, as on master" + else + bad "[$layout] a model with no calls was refused" "rc=$rc $(out)" + fi + + # --- 3. refs present but kind-less: the label is not evidence --------------------------------- + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" full kindless + rc="$(sweep "$P")" + if [ "$rc" = 2 ] && grep -q "carry no RefKind" "$WORK/out" && grep -q "build_mode 'full'" "$WORK/out" \ + && grep -q "(none)=2" "$WORK/out"; then + ok "[$layout] kind-less refs FAULT, naming the mode and per-kind counts ((none)=2)" + else + bad "[$layout] a catalog with untyped edges was not refused by what it holds" "rc=$rc $(out)" + fi + + # --- 4. empty refs still FAULTs --------------------------------------------------------------- + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" fast empty + rc="$(sweep "$P")" + if [ "$rc" = 2 ] && grep -q "^FAULT:" "$WORK/out"; then + ok "[$layout] empty refs still FAULTs (never a clean sweep)" + else + bad "[$layout] empty refs not refused" "rc=$rc $(out)" + fi + + # --- 4b. typed refs, but none the orphan query reads: sweep, and say the orphan list is moot ---- + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" full otherkind + rc="$(sweep "$P")" + if [ "$rc" = 0 ] && grep -q "WARNING: the catalog has no call/action/menu_item/show_page/datasource edges" "$WORK/out" \ + && grep -q "retrieve=1" "$WORK/out"; then + ok "[$layout] no orphan-read kinds: sweeps with the orphan-list WARNING (retrieve=1 shown)" + else + bad "[$layout] missing orphan-read kinds not warned about" "rc=$rc $(out)" + fi + P="$(proj $layout)"; mkdb "$(mdir "$P")/.mxcli/catalog.db" full actions + sweep "$P" >/dev/null + ! grep -q "WARNING: the catalog has no call" "$WORK/out" \ + && ok "[$layout] no orphan-list WARNING when an orphan-read kind is present" \ + || bad "[$layout] orphan-list WARNING printed on an action-bearing catalog" "$(out)" + + # --- 5. app-facts: same guard, through a stub mxcli that drops a prepared catalog ------------- + if [ -f "$AF" ]; then + for variant in source-calls full-actions full-kindless; do + P="$(proj $layout)" + case "$variant" in + source-calls) mkdb "$WORK/prep.db" source calls ;; + full-actions) mkdb "$WORK/prep.db" full actions ;; + full-kindless) mkdb "$WORK/prep.db" full kindless ;; + esac + cat > "$WORK/mxcli-stub" < "\$o" +EOF + chmod +x "$WORK/mxcli-stub" + ( cd "$P" && PROJECT_ROOT="$P" MXCLI="$WORK/mxcli-stub" bash "$AF" --skip-loops --facts-dir "$P/facts" ) > "$WORK/out" 2>&1 + rc=$? + case "$variant" in + source-calls) + if grep -q "catalog ok: build_mode=source" "$WORK/out" && ! grep -q "build_mode is 'source'" "$WORK/out"; then + ok "[$layout] app-facts accepts a 'source' catalog with call edges" + else + bad "[$layout] app-facts refused a 'source' catalog with call edges" "rc=$rc $(out)" + fi ;; + full-actions) + if grep -q "catalog ok: build_mode=full" "$WORK/out"; then + ok "[$layout] app-facts accepts an action-only catalog (zero calls)" + else + bad "[$layout] app-facts refused a model with no calls" "rc=$rc $(out)" + fi ;; + full-kindless) + if [ "$rc" = 2 ] && grep -q "carry no RefKind" "$WORK/out" && grep -q "build_mode 'full'" "$WORK/out" \ + && grep -q "(none)=2" "$WORK/out"; then + ok "[$layout] app-facts FAULTs on kind-less refs, naming the mode and counts" + else + bad "[$layout] app-facts wrote facts from untyped refs" "rc=$rc $(out)" + fi ;; + esac + done + else + bad "app-facts.sh not beside $SUT" + fi +done + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-conformance-per-brd-ledger.sh b/tests/wave2/test-conformance-per-brd-ledger.sh new file mode 100755 index 00000000..a18ac7d6 --- /dev/null +++ b/tests/wave2/test-conformance-per-brd-ledger.sh @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# Usage: bash test-conformance-per-brd-ledger.sh [path-to-conformance-check.sh] +# +# Fixture for project-bin/conformance-check.sh on a per-BRD coverage-ledger layout: +# architecture/modules//coverage-ledger.md an INDEX (no ledger table) +# architecture/modules//coverage-ledger/.md the rows, one file per BRD +# +# The defect (issue #260, field report 2026-10-11, mxcli v0.24.0, four modules): the script read +# only the index and ended every such run in (verbatim from the report — the golden input): +# FAULT: 0 measurable rows extracted from 1 ledger(s). +# while LEDGER_FILE= measured 128 rows. The fix measures every per-BRD file, +# writes one report per BRD, documents LEDGER_FILE, and names an index as an index. +# +# Cases, each on BOTH layouts (single-tree, and two-tree with the .mpr under app/): +# 1. --module Orders: both per-BRD files measured, one report each, the index not read +# 2. two BRDs reuse the same JSON pointer; the baseline keeps both rows (BRD-keyed) +# 3. a per-BRD row going absent is a regression naming its BRD +# 4. no --module: same rows, the index is not measured as a third ledger +# 5. LEDGER_FILE= says "index" and names the directory, not "0 measurable rows" +# 6. a module whose index links to another module's per-BRD file: the link is named +# 7. LEDGER_FILE that does not exist FAULTs instead of falling back +# 8. --help documents LEDGER_FILE +# 9. a BRD file name with a space stays one BRD (one report, not two) +# +# The ledgers and the mxcli stub are SYNTHETIC, as in test-conformance-baseline-scope.sh: the +# field-proof rule's captured-golden-input clause governs parsers of tool output, and nothing +# here exercises the DESCRIBE-output parser — the stub answers the two shapes the script already +# classifies (text + rc 0 = PRESENT, "Error: … not found" + rc 1 = ABSENT). The ledger table +# shape is the 7-column one coverage-ledger.md specifies and the existing fixture uses. + +set -uo pipefail + +SUT="${1:-}" +[ -z "$SUT" ] && SUT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/project-bin/conformance-check.sh" +if [ ! -f "$SUT" ]; then + echo "SKIP: subject not found at $SUT" + echo "SCORE: 0/0 — nothing to test" + exit 0 +fi +SUT="$(cd "$(dirname "$SUT")" && pwd)/$(basename "$SUT")" +[ -f "$(dirname "$SUT")/_common.sh" ] || { echo "FAIL — _common.sh not beside $SUT"; exit 1; } + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok — $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL — $1"; [ -n "${2:-}" ] && echo " got: $2"; } + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/conf-perbrd.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT +GOLDEN='FAULT: 0 measurable rows extracted from 1 ledger(s).' + +ledger_rows() { # ledger_rows + cat > "$1" < -> sets P + P="$WORK/$1" + rm -rf "$P" + local O="$P/architecture/modules/Orders" B="$P/architecture/modules/Billing" + mkdir -p "$O/coverage-ledger" "$B" + if [ "$1" = two ]; then mkdir -p "$P/app"; : > "$P/app/Shop.mpr"; else : > "$P/Shop.mpr"; fi + printf 'Orders.Order\nOrders.ACT_Order_Save\nOrders.Invoice\nOrders.ACT_Invoice_Send\n' > "$P/PRESENT" + cat > "$P/mxcli" <<'EOF' +#!/usr/bin/env bash +cmd=""; while [ $# -gt 0 ]; do [ "$1" = -c ] && cmd="$2"; shift; done +name="${cmd##* }" +d="$(dirname "$0")" +if grep -qx "$name" "$d/PRESENT"; then echo "-- $name"; echo "create $name;"; exit 0; fi +echo "Error: $name not found"; exit 1 +EOF + chmod +x "$P/mxcli" + mkdir -p "$P/architecture/modules/Shipping/coverage-ledger" + ledger_rows "$P/architecture/modules/Shipping/coverage-ledger/F003 shipping.md" Shipping Parcel ACT_Parcel_Send + printf 'Shipping.Parcel\nShipping.ACT_Parcel_Send\n' >> "$P/PRESENT" + ledger_rows "$O/coverage-ledger/F001-orders.md" Orders Order ACT_Order_Save + ledger_rows "$O/coverage-ledger/F002-invoicing.md" Orders Invoice ACT_Invoice_Send + cat > "$O/coverage-ledger.md" <<'EOF' +# Orders — coverage ledger index + +Rows live one file per BRD: +- [F001](coverage-ledger/F001-orders.md) +- [F002](coverage-ledger/F002-invoicing.md) +EOF + cat > "$B/coverage-ledger.md" <<'EOF' +# Billing — coverage ledger index + +Billing has no ledger of its own; its rows are in [F002](../Orders/coverage-ledger/F002-invoicing.md). +EOF +} +run() { ( cd "$P" && PROJECT_ROOT="$P" bash "$SUT" --quiet "$@" ) > "$WORK/out" 2>&1; echo $?; } +out() { tr '\n' ' ' < "$WORK/out" | cut -c1-400; } +base_rows() { grep -c . "$P/docs/conformance/baseline.tsv" 2>/dev/null || echo 0; } + +for layout in single two; do + echo "== $layout-tree layout" + setup "$layout" + + # --- 1. --module measures every per-BRD file ------------------------------------------------ + rc="$(run --module Orders)" + if grep -qF "$GOLDEN" "$WORK/out"; then + bad "[$layout] per-BRD layout read as the index (issue #260 golden FAULT line)" "rc=$rc $(out)" + elif [ "$rc" = 0 ] && grep -q "measured 4 " "$WORK/out"; then + ok "[$layout] --module Orders measures both per-BRD ledgers (4 rows)" + else + bad "[$layout] per-BRD rows not measured" "rc=$rc $(out)" + fi + n=$(ls "$P"/docs/conformance/report-*-F001-orders.tsv "$P"/docs/conformance/report-*-F002-invoicing.tsv 2>/dev/null | grep -c .) + [ "$n" = 2 ] && ok "[$layout] one report per BRD (report--F001-orders.tsv, -F002-invoicing.tsv)" \ + || bad "[$layout] expected 2 per-BRD reports" "$(ls "$P/docs/conformance" 2>/dev/null | tr '\n' ' ')" + + # --- 2. same pointer in two BRDs: two baseline rows, not one ---------------------------------- + [ "$(base_rows)" = 4 ] && grep -q 'F001-orders:`/microflows/0`' "$P/docs/conformance/baseline.tsv" \ + && grep -q 'F002-invoicing:`/microflows/0`' "$P/docs/conformance/baseline.tsv" \ + && ok "[$layout] baseline keys rows by BRD (4 rows, /microflows/0 twice)" \ + || bad "[$layout] baseline collapsed per-BRD rows" "$(cat "$P/docs/conformance/baseline.tsv" 2>/dev/null | tr '\n' ' ')" + + # --- 3. a per-BRD row going absent is a regression -------------------------------------------- + sed -i.bak '/^Orders.ACT_Invoice_Send$/d' "$P/PRESENT" + rc="$(run --module Orders)" + [ "$rc" = 1 ] && grep -q 'F002-invoicing:`/microflows/0`.*was OK, now STALE' "$WORK/out" \ + && ok "[$layout] F002's microflow going absent fails the run, naming the BRD" \ + || bad "[$layout] per-BRD regression not reported" "rc=$rc $(out)" + mv "$P/PRESENT.bak" "$P/PRESENT" + + # --- 4. unscoped run: the index is not a third ledger ----------------------------------------- + rc="$(run)" + if [ "$rc" = 0 ] && grep -q "measured 6 " "$WORK/out" && ! grep -q "Orders/coverage-ledger.md" "$WORK/out"; then + ok "[$layout] unscoped run measures the 6 per-BRD rows (3 BRDs); Orders' index is not read" + else + bad "[$layout] unscoped run wrong on per-BRD layout" "rc=$rc $(out)" + fi + + # --- 5. LEDGER_FILE pointed at the index says so --------------------------------------------- + rc="$( ( cd "$P" && PROJECT_ROOT="$P" LEDGER_FILE=architecture/modules/Orders/coverage-ledger.md \ + bash "$SUT" --quiet --module Orders ) > "$WORK/out" 2>&1; echo $? )" + [ "$rc" = 2 ] && grep -q "is an index" "$WORK/out" && grep -q "architecture/modules/Orders/coverage-ledger/" "$WORK/out" \ + && ! grep -qF "$GOLDEN" "$WORK/out" \ + && ok "[$layout] LEDGER_FILE= is named an index and points at its directory" \ + || bad "[$layout] index reported as '0 measurable rows'" "rc=$rc $(out)" + + # --- 6. an index that links to another module's per-BRD file names the link ------------------ + rc="$(run --module Billing)" + [ "$rc" = 2 ] && grep -q "is an index" "$WORK/out" && grep -q "Orders/coverage-ledger/F002-invoicing.md" "$WORK/out" \ + && ok "[$layout] Billing's index names the ledger its rows are in" \ + || bad "[$layout] cross-module index not explained" "rc=$rc $(out)" + + # --- 9. a BRD id with a space is one BRD -------------------------------------------------------- + rm -f "$P"/docs/conformance/report-*.tsv + rc="$(run --module Shipping)" + n=$(ls "$P"/docs/conformance/ | grep -c '^report-') + [ "$rc" = 0 ] && [ "$n" = 1 ] && ls "$P"/docs/conformance/report-*-"F003 shipping.tsv" >/dev/null 2>&1 \ + && ok "[$layout] 'F003 shipping.md' writes one report, report--F003 shipping.tsv" \ + || bad "[$layout] a BRD id with a space was split" "rc=$rc $(ls "$P/docs/conformance" | tr '\n' '|')" + + # --- 7. a LEDGER_FILE that does not exist FAULTs ---------------------------------------------- + rc="$( ( cd "$P" && PROJECT_ROOT="$P" LEDGER_FILE=architecture/nope.md bash "$SUT" --quiet --module Orders ) \ + > "$WORK/out" 2>&1; echo $? )" + [ "$rc" = 2 ] && grep -q "LEDGER_FILE=architecture/nope.md" "$WORK/out" \ + && ok "[$layout] missing LEDGER_FILE FAULTs instead of falling back" \ + || bad "[$layout] missing LEDGER_FILE silently fell back" "rc=$rc $(out)" +done + +# --- 8. LEDGER_FILE is in the usage text -------------------------------------------------------- +( cd "$P" && PROJECT_ROOT="$P" bash "$SUT" --help ) 2>/dev/null | grep -q "LEDGER_FILE" && ok "--help documents LEDGER_FILE" \ + || bad "--help does not mention LEDGER_FILE" + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-review-conformance-per-brd.sh b/tests/wave2/test-review-conformance-per-brd.sh new file mode 100755 index 00000000..6e806a48 --- /dev/null +++ b/tests/wave2/test-review-conformance-per-brd.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# Usage: bash test-review-conformance-per-brd.sh [path-to-review-module.sh] +# +# Fixture for project-bin/review-module.sh's conformance rung on a per-BRD ledger. +# +# The defect (review of the #260 fix, 2026-10-13): conformance-check.sh writes one report per +# BRD (docs/conformance/report--.tsv) for a per-BRD ledger, and no report-.tsv. +# review-module.sh read exactly one file on both of its paths: +# - --reuse-conformance docs/conformance/report-.tsv (what verify-module.sh passes): +# the file does not exist, so rung 13 of verify-module FAULTed "does not exist"; +# - its own run: `ls -t report-*.tsv | head -1`, i.e. one BRD's rows, the others dropped. +# Both now read every report of the run and slice this module's rows out of all of them. +# +# Cases, each on BOTH layouts (single-tree, and two-tree with the .mpr under app/): +# 1. --reuse-conformance : both per-BRD siblings are read (2 Orders rows, not a FAULT) +# 1b. a module with ONLY the per-BRD directory (no coverage-ledger.md index) is a ledger, not +# "no ledger"; and the reuse log field names the files read, not the absent base name +# 2. its own run (stub conformance-check.sh in /bin, which _tool prefers): rows from +# both per-BRD reports reach conformance.tsv +# +# The reports and the mxcli / conformance-check stubs are SYNTHETIC: the report columns are the +# six conformance-check.sh writes, and the assertions are about which files are read. Later +# rungs (graph sweep, coverage) fault harmlessly on the empty fixture project; only the +# conformance slice is asserted. + +set -uo pipefail + +SUT="${1:-}" +[ -z "$SUT" ] && SUT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/project-bin/review-module.sh" +if [ ! -f "$SUT" ]; then + echo "SKIP: subject not found at $SUT" + echo "SCORE: 0/0 — nothing to test" + exit 0 +fi +SUT="$(cd "$(dirname "$SUT")" && pwd)/$(basename "$SUT")" +[ -f "$(dirname "$SUT")/_common.sh" ] || { echo "FAIL — _common.sh not beside $SUT"; exit 1; } + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok — $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL — $1"; [ -n "${2:-}" ] && echo " got: $2"; } + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/review-perbrd.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT +D="$(date +%Y-%m-%d)" + +report() { # report + { printf 'module\tpointer\tstored\tobserved\tverdict\tcommand\n' + printf 'Orders\t%s:`/microflows/0`\tbuilt\tPRESENT\tOK\tDESCRIBE MICROFLOW Orders.X\n' "$2" + printf 'Billing\t%s:`/microflows/1`\tbuilt\tPRESENT\tOK\tDESCRIBE MICROFLOW Billing.Y\n' "$2" + } > "$1" +} + +setup() { # setup -> sets P + P="$WORK/$1"; rm -rf "$P" + mkdir -p "$P/architecture/modules/Orders/coverage-ledger" "$P/docs/conformance" "$P/bin" + if [ "$1" = two ]; then mkdir -p "$P/app"; : > "$P/app/Shop.mpr"; else : > "$P/Shop.mpr"; fi + printf '# Orders — index\n- [F001](coverage-ledger/F001.md)\n' > "$P/architecture/modules/Orders/coverage-ledger.md" + : > "$P/architecture/modules/Orders/coverage-ledger/F001.md" + cat > "$P/mxcli" <<'EOF' +#!/usr/bin/env bash +case "$*" in *"SHOW MODULES"*) echo "| Orders |"; exit 0 ;; esac +echo "Error: not available in this fixture"; exit 1 +EOF + chmod +x "$P/mxcli" + # Module-exists probe: the catalog answers instantly; without it review-module falls back to + # `mxcli SHOW MODULES` under a 60 s watchdog it waits out in full. + if command -v sqlite3 >/dev/null 2>&1; then + mkdir -p "$P/.mxcli" + printf "CREATE TABLE objects (QualifiedName TEXT, ModuleName TEXT);\nINSERT INTO objects VALUES ('Orders.X','Orders');\n" \ + | sqlite3 "$P/.mxcli/catalog.db" + fi +} +run() { ( cd "$P" && PROJECT_ROOT="$P" MXTK_ROOT="" bash "$SUT" Orders --json-only --out "$P/out/report.json" "$@" ) \ + > "$WORK/out" 2>&1; echo $?; } +slice() { awk -F'\t' 'NR>1 && $1=="Orders"' "$P/.claude/loop/review/Orders/conformance.tsv" 2>/dev/null | grep -c . ; } +out() { sed $'s/\033\[[0-9;]*m//g' "$WORK/out" | tr '\n' ' ' | cut -c1-400; } + +for layout in single two; do + echo "== $layout-tree layout" + + # --- 1. reuse: verify-module passes the base name; the per-BRD siblings are the run --------- + setup "$layout" + report "$P/docs/conformance/report-$D-F001.tsv" F001 + report "$P/docs/conformance/report-$D-F002.tsv" F002 + run --reuse-conformance "docs/conformance/report-$D.tsv" >/dev/null + if grep -q "does not exist" "$WORK/out"; then + bad "[$layout] reuse FAULTed on a per-BRD run (base report-.tsv absent)" "$(out)" + elif [ "$(slice)" = 2 ] && grep -q "2 rows from 2 report(s)" "$WORK/out"; then + ok "[$layout] reuse reads both per-BRD reports (2 Orders rows, Billing's filtered out)" + else + bad "[$layout] reuse did not slice both per-BRD reports" "slice=$(slice) $(out)" + fi + + # --- 1b. directory-only ledger; log field names the files actually read ---------------------- + setup "$layout" + rm -f "$P/architecture/modules/Orders/coverage-ledger.md" + report "$P/docs/conformance/report-$D-F001.tsv" F001 + report "$P/docs/conformance/report-$D-F002.tsv" F002 + run --reuse-conformance "docs/conformance/report-$D.tsv" >/dev/null + S="$P/.claude/loop/review/Orders/summary.tsv" + if grep -q "no architecture/modules/Orders/coverage-ledger.md" "$WORK/out"; then + bad "[$layout] directory-only ledger reported as no ledger" "$(out)" + elif [ "$(slice)" = 2 ] && grep -q "conformance (reused).PASS.*report-$D-F001.tsv, docs/conformance/report-$D-F002.tsv" "$S" \ + && ! grep -q "PASS.0.-.docs/conformance/report-$D.tsv\$" "$S"; then + ok "[$layout] directory-only ledger measured; summary names report-$D-F001/F002.tsv" + else + bad "[$layout] directory-only ledger or log field wrong" "slice=$(slice) $(tr '\n' ' ' < "$S" 2>/dev/null)" + fi + + # --- 2. own run: every report this run wrote, not just the newest --------------------------- + setup "$layout" + cat > "$P/bin/conformance-check.sh" </dev/null + if [ "$(slice)" = 2 ]; then + ok "[$layout] own run slices rows from both per-BRD reports (2 Orders rows)" + else + bad "[$layout] own run read only the newest per-BRD report" "slice=$(slice) $(out)" + fi +done + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-verify-ledger-hint.sh b/tests/wave2/test-verify-ledger-hint.sh new file mode 100755 index 00000000..41d1cb3f --- /dev/null +++ b/tests/wave2/test-verify-ledger-hint.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Usage: bash test-verify-ledger-hint.sh [path-to-verify-module.sh] +# +# Fixture for project-bin/verify-module.sh's ledger hint (resolve_ledger_hint), which the +# coverage rung passes on when coverage-preflight.sh is absent. +# +# The defect (review of #307, 2026-10-13): LEDGER_FILE is documented as a file OR a per-BRD +# directory, but verify-module tested it with `[ -f ]`, so a directory silently fell back to +# architecture/coverage-ledger.md — another ledger, measured as this module's. A module keeping +# only /coverage-ledger/ (no index file) had the same fall-back. +# +# The function is extracted from the subject and called directly (a full verify-module run +# needs a live app). Paths are a scratch tree; both layouts resolve ROOT the same way here, +# because the ledger lives under architecture/ at the project root in both. + +set -uo pipefail + +SUT="${1:-}" +[ -z "$SUT" ] && SUT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/project-bin/verify-module.sh" +if [ ! -f "$SUT" ]; then + echo "SKIP: subject not found at $SUT" + echo "SCORE: 0/0 — nothing to test" + exit 0 +fi + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok — $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL — $1"; [ -n "${2:-}" ] && echo " got: $2"; } + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/verify-ledger.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +FN="$(awk '/^resolve_ledger_hint\(\) \{/{on=1} on{print} on && /^\}/{exit}' "$SUT")" +[ -n "$FN" ] || { bad "no resolve_ledger_hint() in $SUT — LEDGER_FILE is tested with [ -f ] only"; echo; echo "PASS=$PASS FAIL=$FAIL"; exit 1; } + +R="$WORK/proj"; M="$R/architecture/modules" +mkdir -p "$M/Orders/coverage-ledger" "$M/Billing" +: > "$M/Orders/coverage-ledger/F001.md" +: > "$M/Billing/coverage-ledger.md" +: > "$R/architecture/coverage-ledger.md" +hint() { ( ROOT="$R"; MODULE="$1"; LEDGER_FILE="${2:-}"; [ -n "$LEDGER_FILE" ] || unset LEDGER_FILE + eval "$FN"; resolve_ledger_hint ); } + +v="$(hint Orders "$M/Orders/coverage-ledger")" +[ "$v" = "$M/Orders/coverage-ledger" ] && ok "LEDGER_FILE= is kept, not replaced by the project ledger" \ + || bad "directory LEDGER_FILE fell back" "$v" +v="$(hint Orders)" +[ "$v" = "$M/Orders/coverage-ledger" ] && ok "a module with only the per-BRD directory resolves to it" \ + || bad "directory-only module fell back" "$v" +v="$(hint Billing)" +[ "$v" = "$M/Billing/coverage-ledger.md" ] && ok "a module with coverage-ledger.md still resolves to the file" \ + || bad "file ledger changed" "$v" +v="$(hint Shipping)" +[ "$v" = "$R/architecture/coverage-ledger.md" ] && ok "a module with neither falls back to the project ledger, as before" \ + || bad "project fallback changed" "$v" + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/tests/wave2/test-verify-stack-blocked.sh b/tests/wave2/test-verify-stack-blocked.sh new file mode 100755 index 00000000..61f092e4 --- /dev/null +++ b/tests/wave2/test-verify-stack-blocked.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Usage: bash test-verify-stack-blocked.sh [path-to-verify-module.sh] +# +# Fixture for project-bin/verify-module.sh's verdict for the stack rung. +# +# The defect (review of #313, 2026-10-13): test-stack-up.sh exits 4 for "instrument blocked — +# the port `mxcli run --local` needs is held by another process", and its header says to +# record that as blocked, not as a test failure. verify-module's verdict table mapped every +# non-0, non-2 gate exit to FINDING, so a held port read as "the module has findings". It is +# now a FAULT (the instrument did not measure), scoped to the stack rung: exit 4 means NOT +# APPLICABLE from conformance-check.sh, so it is deliberately not a global remap. +# +# The verdict function is extracted from the subject and called directly — a full verify-module +# run needs a live app. The exit codes are the ones test-stack-up.sh's own header documents. + +set -uo pipefail + +SUT="${1:-}" +[ -z "$SUT" ] && SUT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/project-bin/verify-module.sh" +if [ ! -f "$SUT" ]; then + echo "SKIP: subject not found at $SUT" + echo "SCORE: 0/0 — nothing to test" + exit 0 +fi + +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok — $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL — $1"; [ -n "${2:-}" ] && echo " got: $2"; } + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/verify-stack.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +FN="$(awk '/^_report_result\(\) \{/{on=1} on{print} on && /^\}/{exit}' "$SUT")" +[ -n "$FN" ] || { echo "FAIL — no _report_result() in $SUT"; exit 1; } + +# verdict -> the verdict column _report_result writes to the summary +verdict() { + ( ROOT="$WORK"; SUMMARY="$WORK/summary.tsv"; FINDINGS=0; FAULTED=0 + c_ok() { :; }; c_bad() { :; }; c_warn() { :; } + : > "$SUMMARY"; echo "log" > "$WORK/x.log" + eval "$FN" + _report_result "rung" "$WORK/x.log" "$1" 60 "$2" 1 >/dev/null 2>&1 + cut -f2 "$SUMMARY" ) +} + +v="$(verdict stack 4)" +[ "$v" = FAULT ] && ok "stack rung exit 4 (port held) is FAULT, not FINDING" \ + || bad "stack rung exit 4 graded as a test result" "$v" +v="$(verdict stack 1)" +[ "$v" = FINDING ] && ok "stack rung exit 1 (not up) is still FINDING" || bad "stack exit 1 changed" "$v" +v="$(verdict stack 0)" +[ "$v" = PASS ] && ok "stack rung exit 0 is PASS" || bad "stack exit 0 changed" "$v" +v="$(verdict stack 2)" +[ "$v" = FAULT ] && ok "stack rung exit 2 is FAULT" || bad "stack exit 2 changed" "$v" +v="$(verdict gate 4)" +[ "$v" = FINDING ] && ok "exit 4 from a non-stack gate rung is not remapped (scoped fix)" \ + || bad "exit 4 remapped globally" "$v" + +grep -qE 'run "stack \(test-stack-up --check\)" "[^"]*" stack ' "$SUT" \ + && ok "the stack rung is run with kind 'stack'" \ + || bad "the stack rung does not use kind 'stack', so exit 4 still reads as FINDING" + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ]