Skip to content

process(testing): per-project test strategy; affected tests per change, full suite at deploy-ready #409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MendixMau wants to merge 1 commit into from
Closed

process(testing): per-project test strategy; affected tests per change, full suite at deploy-ready #409

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m

## Unreleased

- process(testing): **each project now picks its test strategy with the developer at CAC-5, and per change the default is the gate plus the affected unit tests and journeys only; the full e2e and the full unit suite run at deploy-ready.** New `project-bin/affected-tests.sh <script.mdl>` computes the subject set `testing-shape.md` §3 only described: the elements the script writes (new `exec.sh --subjects`), their transitive callers from ONE batched `SHOW CALLERS OF ... TRANSITIVE` call (pages included on mxcli v0.26.0), then the `.test.mdl` files and `journeys/<Module>.journey.json` files they owe, scoped by the PROJECT.md `## Test strategy` row (`none`/`affected`/`module`/`full`, defaults when absent). It prints run lines, never runs a model-writing test, and says "callers NOT MEASURED" when it has no model. `exec.sh` now prints and logs per-step seconds (`steps: guards check snapshot baseline exec gate lint score`). testing-shape.md, iterative-build-loop.md, existing-app-change.md and conversion-runbook.md drop "the full suite at module close" for "the module's tests at module close, everything at deploy-ready"; baseline words net lower. Field run: a five-app lab project (Mendix 11.15, mxcli v0.26.0), where one exec is 28 to 41 s and a full `mxcli test` about 6 min; three of its real change scripts selected 0 to 1 of 20 journey files each (first call 19 s, later 3 s). Golden callers output captured from mxcli v0.26.0 on a scratch copy of an empty 11.12 app (`tests/wave2/fixtures/affected-tests/CAPTURE.txt`) — the maintainer's decision, 2026-10-16
- fix(install-manifest): **a clean toolkit checkout passes its own `project-bin/` manifest self-check, so adopters no longer see the "named by NEITHER list" warning at the top of every sync.** `coverage-check-all.sh` (found by `review-module.sh` through its `_tool` lookup in the toolkit clone) and `ts-sync.sh` (UNPROVEN and SUPERSEDED per its header) join `MXTK_PROJECT_BIN_NOINSTALL` with reasons. New `tests/wave2/test-manifest-clean-checkout.sh` fails, rather than warns, when a `project-bin/` file is in neither list. — joshuamoesa (#381)
- fix(sync-project): **the "Baseline-only skills cited" warning no longer fires on the toolkit's own wiring block, and no longer sizes a section that does not exist.** The check grepped all of `CLAUDE.md` for `skills/learned-mdl-preflight.md` / `skills/query-the-model.md`, and `wire-agents.sh`'s `mxtk:wiring` block cites the first, so every sync flagged the toolkit's own text; with no "mxcli-project-toolkit Integration" heading it then printed a whole-file word count as "heading to EOF". The wiring block is now stripped before the grep, and the warning is raised only when the heading exists. `tests/wave2/test-sync-dup-heading-exit.sh` T4 flips (no heading: no warning) and gains T7-T9. — joshuamoesa (#382)
- learn(workflow, rest): **the v0.20-era workflow notes are marked superseded for mxcli v0.26.0:** `WAIT FOR NOTIFICATION` takes a name, `NOTIFY WORKFLOW` needs `TARGET Module.Workflow.Element` (CE0166 / MDL-WF16 without), mxcli writes the end-of-boundary-path marker itself so a boundary terminator is no longer a hand-add (the parallel-split terminator is a separate node and unchanged), and published REST `Authentication: (basic)` works (BUG-152 item 1: 401 / 401 / 200 proven, URL includes the resource segment). Sources: a five-app lab field run (Mendix 11.15, mxcli v0.26.0) and mxcli's public `write-workflows` skill and docs. `wf-add-path-terminators.py` already takes an `.mpr` since #294 (the reported crash was the older script). — TK-019 of a five-app lab report
Expand Down
2 changes: 1 addition & 1 deletion bin/lib/install-manifest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ MXTK_AGENTS_STAGE_BUILD="mdl-agent.md gate-agent.md test-agent.md review-agent.m
# may run itself (stops only the pid run --local recorded for THIS project), and `mxcli test` on a
# scratch copy under the test-port lock. exec.sh names app-restart.sh in its watch advisory, so
# it has to be installed beside it.
MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh"
MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh affected-tests.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh"

# Files in project-bin/ that are deliberately NOT installed into projects. The reverse check
# below flags anything named by NEITHER list, so a new file in project-bin/ has to be either
Expand Down
218 changes: 218 additions & 0 deletions project-bin/affected-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
#!/usr/bin/env bash
# affected-tests.sh: which unit tests and journeys a change owes, per the project's test strategy.
#
# ./bin/affected-tests.sh [--moment change|module|deploy] [--module <M>]... [--depth N]
# [--no-model] <script.mdl>...
#
# Prints, one row per line, tab-separated:
# strategy <moment> unit=<scope> journeys=<scope> (<where the scope came from>)
# subject <kind> <Module.Name> line N of <script>
# caller <Module.Name> depth N (SHOW CALLERS OF ... TRANSITIVE)
# callers measured | NOT MEASURED (<reason>)
# unit <file.test.mdl> <why>
# journey <file.journey.json> <why>
# run <the command to run that selection>
# Exit: 0 printed (an empty selection is printed as "none", never as silence) · 2 usage.
#
# WHY (2026-10-16). testing-shape.md §3 defined the per-change "subject set" (changed microflows
# plus their callers, the pages the change reaches) in prose only; exec.sh printed a lane and
# nothing computed the set. Measured in a five-app lab project (Mendix 11.15, mxcli v0.26.0):
# one exec 28-41 s, a full mxcli test about 6 min, conformance 645 s, coverage 377 s. A change
# that owes three tests was paying for the whole suite whenever an agent chose "to be safe".
# The maintainer's decision: per change, gate + affected tests only; the full e2e and the full
# unit suite run at deploy-ready, and each project picks its strategy with the developer (CAC-5).
#
# THE STRATEGY. PROJECT.md "## Test strategy" (skills/checkpoints/checkpoint-build.md writes it):
# | Moment | Unit tests | Journeys | Status |
# | Every change | affected | affected | CONFIRMED |
# | Module close | module | module | CONFIRMED |
# | Deploy-ready | full | full | CONFIRMED |
# Scopes: none | affected | module | full. No section, or no row for the moment: these defaults,
# and the strategy row says "default". --moment picks the row (change is the default).
#
# HOW THE SELECTION IS MADE (judgement-free facts only; skills-over-scripts.md):
# 1. Subjects: `bin/exec.sh --subjects <script>` (the same statement lexer as the change class).
# 2. Callers: ONE batched `mxcli -p <app>.mpr -c "SHOW CALLERS OF X TRANSITIVE; ..."` for every
# microflow, nanoflow, page, snippet and layout subject. Golden input: a verbatim v0.26.0
# capture (tests/wave2/fixtures/affected-tests/CAPTURE.txt). Transitive callers include pages
# (a button's CALL MICROFLOW, a SHOW PAGE) on v0.26.0. The first call builds mxcli's catalog
# (~40 s on a 275-flow app), later ones ~4 s. --no-model, no project mxcli, no .mpr, or output
# that does not parse: "callers NOT MEASURED", and the selection is the subjects alone.
# 3. affected unit: *.test.mdl files that name a subject, or a caller at depth <= --depth
# (default 1: "the tests of its callers"). Module, entity, association, enumeration and
# access subjects select every test file that names their module (testing-shape.md §3 table).
# 4. affected journeys: journeys/<M>.journey.json for every module M owning a subject or ANY
# transitive caller (a page reached at depth 4 is still a page the change reaches), plus any
# journey file (journeys/ or tests/e2e/) that names a subject literally.
# 5. module: every test file naming the module, journeys/<M>.journey.json; the modules are the
# subjects' modules plus every --module. full: every test file, every journey file.
# What it does NOT do: run anything. `mxcli test` writes to the model, so the run rows go through
# bin/test-scratch.sh (a scratch copy) and the journey runner; the caller runs them.
#
# Producer: exec.sh prints "affected tests: ./bin/affected-tests.sh <script>" on every MDL run.
# Overrides: JOURNEY_DIR (default journeys/), MXCLI (default the project's ./mxcli), MPR_FILE.
# Bash 3.2 / BSD awk / Git Bash.
set -uo pipefail
. "$(dirname "$0")/_common.sh"
HERE="$(cd "$(dirname "$0")" && pwd)"

MOMENT="change"; DEPTH=1; NO_MODEL=0; MODULES=""; SCRIPTS=""
usage() { sed -n '2,15p' "$0" >&2; exit 2; }
while [ $# -gt 0 ]; do
case "$1" in
--moment) MOMENT="${2:-}"; shift 2 ;;
--module) MODULES="$MODULES ${2:-}"; shift 2 ;;
--depth) DEPTH="${2:-}"; shift 2 ;;
--no-model) NO_MODEL=1; shift ;;
-h|--help) usage ;;
--*) echo "✗ unknown option $1" >&2; usage ;;
*) [ -f "$1" ] || { echo "✗ no such script: $1" >&2; exit 2; }
SCRIPTS="$SCRIPTS
$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"; shift ;;
esac
done
case "$MOMENT" in change|module|deploy) ;; *) echo "✗ --moment takes change, module or deploy" >&2; exit 2 ;; esac
case "$DEPTH" in ''|*[!0-9]*) echo "✗ --depth takes a number" >&2; exit 2 ;; esac
SCRIPTS="$(printf '%s\n' "$SCRIPTS" | sed '/^$/d')"
[ -n "$SCRIPTS" ] || [ "$MOMENT" != change ] || { echo "✗ a change needs at least one script" >&2; usage; }

cd "$PROJECT_ROOT" || exit 2
JDIR="${JOURNEY_DIR:-journeys}"
WORK="$(mktemp -d "${TMPDIR:-/tmp}/affected.XXXXXX")"; trap 'rm -rf "$WORK"' EXIT
out() { printf '%s\n' "$*"; }

# ── Strategy row ─────────────────────────────────────────────────────────────
case "$MOMENT" in change) ROW="Every change"; DU=affected; DJ=affected ;;
module) ROW="Module close"; DU=module; DJ=module ;;
deploy) ROW="Deploy-ready"; DU=full; DJ=full ;; esac
U="$DU"; J="$DJ"; FROM="default: PROJECT.md has no '## Test strategy' row '$ROW' (CAC-5 records it)"
if [ -f PROJECT.md ]; then
_row="$(awk -v r="$ROW" '
/^## /{ on = ($0 ~ /^## Test strategy/) ; next }
on && /^\|/ { split($0, c, "|"); x = c[2]; gsub(/^[ \t]+|[ \t]+$/, "", x)
if (tolower(x) == tolower(r)) { u = c[3]; j = c[4]; s = c[5]
gsub(/[ \t`*]/, "", u); gsub(/[ \t`*]/, "", j); gsub(/^[ \t]+|[ \t]+$/, "", s)
print tolower(u) "\t" tolower(j) "\t" s; exit } }' PROJECT.md)"
if [ -n "$_row" ]; then
_u="$(printf '%s' "$_row" | cut -f1)"; _j="$(printf '%s' "$_row" | cut -f2)"; _s="$(printf '%s' "$_row" | cut -f3)"
case "$_u" in none|affected|module|full) U="$_u" ;; *) FROM="row '$ROW' unit scope '$_u' unknown, default used; " ;; esac
case "$_j" in none|affected|module|full) J="$_j" ;; *) FROM="${FROM}row '$ROW' journey scope '$_j' unknown, default used; " ;; esac
case "$FROM" in default:*) FROM="PROJECT.md ## Test strategy, row '$ROW'${_s:+, $_s}" ;; *) FROM="${FROM}PROJECT.md ## Test strategy" ;; esac
fi
fi
out "strategy $MOMENT unit=$U journeys=$J ($FROM)"

# ── Subjects ─────────────────────────────────────────────────────────────────
: > "$WORK/subjects"
if [ -n "$SCRIPTS" ]; then
printf '%s\n' "$SCRIPTS" | while IFS= read -r s; do
bash "$HERE/exec.sh" --subjects "$s" 2>/dev/null | while IFS=' ' read -r k n l; do
printf '%s\t%s\t%s of %s\n' "$k" "$n" "$l" "$(basename "$s")"
done
done > "$WORK/subjects"
fi
while IFS=' ' read -r k n w; do out "subject $k $n $w"; done < "$WORK/subjects"
[ -n "$SCRIPTS" ] && [ ! -s "$WORK/subjects" ] && out "subject none - no model-writing statement in the script(s)"
mod_of() { printf '%s\n' "${1%%.*}"; }
awk -F'\t' '$2 != "-" { split($2, a, "."); print a[1] }' "$WORK/subjects" > "$WORK/modules"
for m in $MODULES; do echo "$m" >> "$WORK/modules"; done
sort -u "$WORK/modules" -o "$WORK/modules"

# ── Callers (one batched mxcli call) ─────────────────────────────────────────
: > "$WORK/callers"
awk -F'\t' '$1 ~ /^(microflow|nanoflow|page|snippet|layout)$/ { print $2 }' "$WORK/subjects" | sort -u > "$WORK/flowish"
CALLERS="NOT MEASURED"
if [ "$U" != affected ] && [ "$J" != affected ]; then
CALLERS="not needed (no affected scope at this moment)"
elif [ ! -s "$WORK/flowish" ]; then
CALLERS="not needed (no microflow, nanoflow, page, snippet or layout subject)"
elif [ "$NO_MODEL" = 1 ]; then
CALLERS="NOT MEASURED (--no-model)"
else
MX="${MXCLI:-$(find_project_mxcli 2>/dev/null || true)}"
MPR="$(find_mpr 2>/dev/null || true)"
if [ -z "$MX" ] || [ ! -x "$MX" ]; then CALLERS="NOT MEASURED (no project mxcli; set MXCLI)"
elif [ -z "$MPR" ]; then CALLERS="NOT MEASURED (no .mpr found)"
else
Q="$(sed 's/.*/SHOW CALLERS OF & TRANSITIVE/' "$WORK/flowish" | awk '{ printf "%s%s", (NR > 1 ? "; " : ""), $0 }')"
"$MX" -p "$MPR" -c "$Q" > "$WORK/mx.out" 2> "$WORK/mx.err"; RC=$?
_blocks="$(grep -c '^Callers of ' "$WORK/mx.out" 2>/dev/null)" || _blocks=0
if [ "$RC" -ne 0 ] || [ "$_blocks" -ne "$(wc -l < "$WORK/flowish" | tr -d ' ')" ]; then
_why="$(grep -v -e '^✓' -e '^WARNING: This is a' -e '^[[:space:]]*$' "$WORK/mx.err" | head -1)"
CALLERS="NOT MEASURED (mxcli exit $RC, $_blocks of $(wc -l < "$WORK/flowish" | tr -d ' ') caller blocks${_why:+: $_why})"
else
CALLERS="measured (SHOW CALLERS OF ... TRANSITIVE, $_blocks subject(s))"
awk '/^\| *[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]* *\| *[0-9]+ *\|$/ {
split($0, c, "|"); n = c[2]; d = c[3]; gsub(/ /, "", n); gsub(/ /, "", d)
if (!(n in best) || d + 0 < best[n]) best[n] = d + 0 }
END { for (n in best) print n "\t" best[n] }' "$WORK/mx.out" | sort -t ' ' -k2,2n -k1,1 > "$WORK/callers"
fi
fi
fi
while IFS=' ' read -r n d; do out "caller $n depth $d"; done < "$WORK/callers"
out "callers $CALLERS"

# ── Candidate files ──────────────────────────────────────────────────────────
find . \( -name .git -o -name .mpr-snapshots -o -name node_modules -o -name deployment \) -prune -o \
-type f -name '*.test.mdl' -print 2>/dev/null | sed 's|^\./||' | sort > "$WORK/tests"
for _d in "$JDIR" tests/e2e; do [ -d "$_d" ] && find "$_d" -type f -name '*.journey.json' 2>/dev/null; done | sed 's|^\./||' | sort -u > "$WORK/journeys"

# names: the strings a test or journey must contain to be selected
names_for_unit() {
awk -F'\t' '$2 != "-" && $1 !~ /^(entity|association|enumeration|access|module)$/ { print $2 }' "$WORK/subjects"
awk -F'\t' -v d="$DEPTH" '$2 + 0 <= d { print $1 }' "$WORK/callers"
}
modules_for_unit() { awk -F'\t' '$2 != "-" && $1 ~ /^(entity|association|enumeration|access|module)$/ { split($2, a, "."); print a[1] }' "$WORK/subjects" | sort -u; }

select_unit() { # $1 = scope; prints "<file>\t<why>"
case "$1" in
none) ;;
full) sed 's/$/ full suite/' "$WORK/tests" ;;
module)
while IFS= read -r m; do [ -n "$m" ] || continue
while IFS= read -r f; do grep -qE "(^|[^A-Za-z0-9_])$m\." "$f" 2>/dev/null && echo "$f names module $m"; done < "$WORK/tests"
done < "$WORK/modules" ;;
affected)
names_for_unit | sort -u > "$WORK/unames"
[ -s "$WORK/unames" ] && while IFS= read -r f; do
_hit="$(grep -o -w -F -f "$WORK/unames" "$f" 2>/dev/null | head -1)"
[ -n "$_hit" ] && echo "$f names $_hit"
done < "$WORK/tests"
modules_for_unit | while IFS= read -r m; do
while IFS= read -r f; do grep -qE "(^|[^A-Za-z0-9_])$m\." "$f" 2>/dev/null && echo "$f names module $m (data or access change)"; done < "$WORK/tests"
done ;;
esac | awk -F'\t' '!seen[$1]++'
}

select_journeys() { # $1 = scope; prints "<file>\t<why>"
case "$1" in
none) ;;
full) sed 's/$/ full suite/' "$WORK/journeys" ;;
module|affected)
{ cat "$WORK/modules"; [ "$1" = affected ] && cut -f1 "$WORK/callers" | sed 's/\..*//'; } | sort -u |
while IFS= read -r m; do
[ -n "$m" ] && [ -f "$JDIR/$m.journey.json" ] && echo "$JDIR/$m.journey.json module $m"
done
if [ "$1" = affected ]; then
awk -F'\t' '$2 ~ /\./ { print $2 }' "$WORK/subjects" | sort -u > "$WORK/jnames"
[ -s "$WORK/jnames" ] && while IFS= read -r f; do
_hit="$(grep -o -w -F -f "$WORK/jnames" "$f" 2>/dev/null | head -1)"
[ -n "$_hit" ] && echo "$f names $_hit"
done < "$WORK/journeys"
fi ;;
esac | awk -F'\t' '!seen[$1]++'
}

select_unit "$U" > "$WORK/unit.sel"
select_journeys "$J" > "$WORK/journey.sel"
if [ -s "$WORK/unit.sel" ]; then sed 's/^/unit /' "$WORK/unit.sel"
else out "unit none scope $U: $( [ "$U" = none ] && echo 'strategy says none' || echo "0 of $(wc -l < "$WORK/tests" | tr -d ' ') .test.mdl file(s) match")"; fi
if [ -s "$WORK/journey.sel" ]; then sed 's/^/journey /' "$WORK/journey.sel"
else out "journey none scope $J: $( [ "$J" = none ] && echo 'strategy says none' || echo "0 of $(wc -l < "$WORK/journeys" | tr -d ' ') journey file(s) in $JDIR/ and tests/e2e/ match")"; fi

# ── Run rows ─────────────────────────────────────────────────────────────────
_uf="$(cut -f1 "$WORK/unit.sel" | tr '\n' ' ' | sed 's/ $//')"
_jf="$(cut -f1 "$WORK/journey.sel" | tr '\n' ' ' | sed 's/ $//')"
[ -n "$_uf" ] && out "run ./bin/test-scratch.sh -- $_uf --local"
[ -n "$_jf" ] && out "run node tests/e2e/journey-runner.js $_jf"
exit 0
Loading
Loading