From cbfb225b101656847180c2715df89150bddcef9e Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 16 Oct 2026 09:56:49 +0800 Subject: [PATCH] process(testing): per-project test strategy; affected tests per change, full suite at deploy-ready Per change the default is now the gate plus the affected unit tests and journeys only. The full e2e and the full unit suite run at deploy-ready, and each project records its own choice with the developer in the existing CAC-5 Offer (PROJECT.md "## Test strategy"). - project-bin/affected-tests.sh (new): subjects from exec.sh --subjects, transitive callers from one batched SHOW CALLERS OF ... TRANSITIVE call, then the .test.mdl and journey files owed at the moment's scope (none / affected / module / full). Prints run lines; never runs a model-writing test; says "callers NOT MEASURED" without a model. - exec.sh: --subjects advisory mode; per-step seconds in the summary and the BUILD-LOG detail; the logic lane no longer says the full suite runs at module close. - testing-shape.md, iterative-build-loop.md, existing-app-change.md, conversion-runbook.md: module close runs the module's tests, deploy-ready runs everything. Baseline 75,225 to 75,221 words. - checkpoint-build.md: the test strategy table in the Stage 5 Offer. Field run: a five-app lab project (Mendix 11.15, mxcli v0.26.0), three real change scripts on a scratch copy: 0 to 1 of 20 journey files selected per change, first call 19 s, later 3 s. Golden callers output captured from mxcli v0.26.0 (fixtures/affected-tests/CAPTURE.txt). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + bin/lib/install-manifest.sh | 2 +- project-bin/affected-tests.sh | 218 ++++++++++++++++++ project-bin/exec.sh | 62 ++++- skills/checkpoints/checkpoint-build.md | 21 ++ skills/conversion-runbook.md | 4 +- skills/existing-app-change.md | 5 +- skills/iterative-build-loop.md | 6 +- skills/testing-shape.md | 18 +- .../wave2/fixtures/affected-tests/CAPTURE.txt | 33 +++ .../wave2/fixtures/affected-tests/callers.err | 57 +++++ .../fixtures/affected-tests/callers.exit | 1 + .../wave2/fixtures/affected-tests/callers.out | 26 +++ .../fixtures/affected-tests/callers.query | 1 + .../wave2/fixtures/affected-tests/noproj.err | 2 + .../wave2/fixtures/affected-tests/noproj.exit | 1 + .../wave2/fixtures/affected-tests/noproj.out | 0 tests/wave2/fixtures/affected-tests/shop.mdl | 43 ++++ .../fixtures/affected-tests/val-order.out | 9 + tests/wave2/test-affected-tests.sh | 176 ++++++++++++++ tests/wave2/test-exec-step-times.sh | 100 ++++++++ 21 files changed, 767 insertions(+), 19 deletions(-) create mode 100755 project-bin/affected-tests.sh create mode 100644 tests/wave2/fixtures/affected-tests/CAPTURE.txt create mode 100644 tests/wave2/fixtures/affected-tests/callers.err create mode 100644 tests/wave2/fixtures/affected-tests/callers.exit create mode 100644 tests/wave2/fixtures/affected-tests/callers.out create mode 100644 tests/wave2/fixtures/affected-tests/callers.query create mode 100644 tests/wave2/fixtures/affected-tests/noproj.err create mode 100644 tests/wave2/fixtures/affected-tests/noproj.exit create mode 100644 tests/wave2/fixtures/affected-tests/noproj.out create mode 100644 tests/wave2/fixtures/affected-tests/shop.mdl create mode 100644 tests/wave2/fixtures/affected-tests/val-order.out create mode 100644 tests/wave2/test-affected-tests.sh create mode 100644 tests/wave2/test-exec-step-times.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index b7202542..c7606427 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- process(testing): **each project now picks its test strategy with the developer at CAC-5, and per change the default is the gate plus the affected unit tests and journeys only; the full e2e and the full unit suite run at deploy-ready.** New `project-bin/affected-tests.sh ` computes the subject set `testing-shape.md` §3 only described: the elements the script writes (new `exec.sh --subjects`), their transitive callers from ONE batched `SHOW CALLERS OF ... TRANSITIVE` call (pages included on mxcli v0.26.0), then the `.test.mdl` files and `journeys/.journey.json` files they owe, scoped by the PROJECT.md `## Test strategy` row (`none`/`affected`/`module`/`full`, defaults when absent). It prints run lines, never runs a model-writing test, and says "callers NOT MEASURED" when it has no model. `exec.sh` now prints and logs per-step seconds (`steps: guards check snapshot baseline exec gate lint score`). testing-shape.md, iterative-build-loop.md, existing-app-change.md and conversion-runbook.md drop "the full suite at module close" for "the module's tests at module close, everything at deploy-ready"; baseline words net lower. Field run: a five-app lab project (Mendix 11.15, mxcli v0.26.0), where one exec is 28 to 41 s and a full `mxcli test` about 6 min; three of its real change scripts selected 0 to 1 of 20 journey files each (first call 19 s, later 3 s). Golden callers output captured from mxcli v0.26.0 on a scratch copy of an empty 11.12 app (`tests/wave2/fixtures/affected-tests/CAPTURE.txt`) — the maintainer's decision, 2026-10-16 - fix(install-manifest): **a clean toolkit checkout passes its own `project-bin/` manifest self-check, so adopters no longer see the "named by NEITHER list" warning at the top of every sync.** `coverage-check-all.sh` (found by `review-module.sh` through its `_tool` lookup in the toolkit clone) and `ts-sync.sh` (UNPROVEN and SUPERSEDED per its header) join `MXTK_PROJECT_BIN_NOINSTALL` with reasons. New `tests/wave2/test-manifest-clean-checkout.sh` fails, rather than warns, when a `project-bin/` file is in neither list. — joshuamoesa (#381) - fix(sync-project): **the "Baseline-only skills cited" warning no longer fires on the toolkit's own wiring block, and no longer sizes a section that does not exist.** The check grepped all of `CLAUDE.md` for `skills/learned-mdl-preflight.md` / `skills/query-the-model.md`, and `wire-agents.sh`'s `mxtk:wiring` block cites the first, so every sync flagged the toolkit's own text; with no "mxcli-project-toolkit Integration" heading it then printed a whole-file word count as "heading to EOF". The wiring block is now stripped before the grep, and the warning is raised only when the heading exists. `tests/wave2/test-sync-dup-heading-exit.sh` T4 flips (no heading: no warning) and gains T7-T9. — joshuamoesa (#382) - learn(workflow, rest): **the v0.20-era workflow notes are marked superseded for mxcli v0.26.0:** `WAIT FOR NOTIFICATION` takes a name, `NOTIFY WORKFLOW` needs `TARGET Module.Workflow.Element` (CE0166 / MDL-WF16 without), mxcli writes the end-of-boundary-path marker itself so a boundary terminator is no longer a hand-add (the parallel-split terminator is a separate node and unchanged), and published REST `Authentication: (basic)` works (BUG-152 item 1: 401 / 401 / 200 proven, URL includes the resource segment). Sources: a five-app lab field run (Mendix 11.15, mxcli v0.26.0) and mxcli's public `write-workflows` skill and docs. `wf-add-path-terminators.py` already takes an `.mpr` since #294 (the reported crash was the older script). — TK-019 of a five-app lab report diff --git a/bin/lib/install-manifest.sh b/bin/lib/install-manifest.sh index 13a78191..f8003662 100644 --- a/bin/lib/install-manifest.sh +++ b/bin/lib/install-manifest.sh @@ -115,7 +115,7 @@ MXTK_AGENTS_STAGE_BUILD="mdl-agent.md gate-agent.md test-agent.md review-agent.m # may run itself (stops only the pid run --local recorded for THIS project), and `mxcli test` on a # scratch copy under the test-port lock. exec.sh names app-restart.sh in its watch advisory, so # it has to be installed beside it. -MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh" +MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh affected-tests.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh" # Files in project-bin/ that are deliberately NOT installed into projects. The reverse check # below flags anything named by NEITHER list, so a new file in project-bin/ has to be either diff --git a/project-bin/affected-tests.sh b/project-bin/affected-tests.sh new file mode 100755 index 00000000..7fc155da --- /dev/null +++ b/project-bin/affected-tests.sh @@ -0,0 +1,218 @@ +#!/usr/bin/env bash +# affected-tests.sh: which unit tests and journeys a change owes, per the project's test strategy. +# +# ./bin/affected-tests.sh [--moment change|module|deploy] [--module ]... [--depth N] +# [--no-model] ... +# +# Prints, one row per line, tab-separated: +# strategy unit= journeys= () +# subject line N of