From 97e740300b8b872e4b9a5b9d779336a009cfe0bd Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 16 Oct 2026 10:40:43 +0800 Subject: [PATCH 1/7] process(testing): per-project test strategy at CAC-5, module close scoped, deploy-ready runs everything (HELD) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + project-bin/exec.sh | 2 +- skills/checkpoints/checkpoint-build.md | 21 +++++++++++ skills/conversion-runbook.md | 4 +- skills/existing-app-change.md | 5 ++- skills/iterative-build-loop.md | 6 +-- skills/testing-shape.md | 18 ++++----- tests/wave2/test-test-strategy-docs.sh | 52 ++++++++++++++++++++++++++ 8 files changed, 93 insertions(+), 16 deletions(-) create mode 100644 tests/wave2/test-test-strategy-docs.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 2334398a..f0955c90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- process(testing): **HELD, do not merge before the test-strategy A/B (E3) and the baseline-text A/B: each project picks its test strategy with the developer at CAC-5, and the default shifts from the full suite at module close to the module's own tests there, everything at deploy-ready.** The CAC-5 Offer in `checkpoint-build.md` shows a three-row table (every change `affected`, module close `module`, deploy-ready `full`) and records it in PROJECT.md `## Test strategy` with a Status column (`ASSUMED` in auto mode). `testing-shape.md`, `iterative-build-loop.md`, `conversion-runbook.md` and the exec.sh logic lane text drop "full suite at module close" for "deploy-ready"; `existing-app-change.md` keeps "every gate is also a regression gate" but runs the baseline journeys of the touched modules per gate and the whole baseline at deploy-ready. Baseline words 75,225 to 75,221. New `tests/wave2/test-test-strategy-docs.sh` keeps the wording and the three defaults from drifting apart (9 of 10 checks fail on master). Split out of #409 — the maintainer's decision, 2026-10-16 - fix(conformance-check): read ledger rows whose pointer cell is bare (no backticks) and treat status `planned` as not-built; a ledger whose acceptance text is also bare prose is still not measured (acceptance must hold a backticked SHOW/DESCRIBE), but it now exits 2 with a FAULT that says rows were found and the text is prose, instead of "shape changed or glob matched nothing" (TK-006, partial: that ledger shape is diagnosed, not fixed) — field report TK-006 - fix(app-restart): **a secret with spaces in quotes is masked whole, and `Pwd` and `Credential` join the secret words.** Follow-up to #403: `--constant "Mod.ApiKey=two words"` and `--db-password 'two words'` printed everything after the first word (the quoted-flag case leaked before #403 too). `mask()` now masks each argument before quoting it for display, so the whole quoted value goes; the executed command is untouched, and a real-launch test proves the started app receives the raw argv; an env assignment at the start of a `--cmd` string keeps the whole command visible; the per-argument seds only run for arguments holding a secret word. The substring match still over-masks names like `Monkey` (Key) and `Turkey`; that is display only, kept, and pinned by a test. `tests/wave2/test-app-restart.sh` gains 22 cases, 8 of them failing on master. — review of #403 - fix(install-manifest): **a clean toolkit checkout passes its own `project-bin/` manifest self-check, so adopters no longer see the "named by NEITHER list" warning at the top of every sync.** `coverage-check-all.sh` (found by `review-module.sh` through its `_tool` lookup in the toolkit clone) and `ts-sync.sh` (UNPROVEN and SUPERSEDED per its header) join `MXTK_PROJECT_BIN_NOINSTALL` with reasons. New `tests/wave2/test-manifest-clean-checkout.sh` fails, rather than warns, when a `project-bin/` file is in neither list. — joshuamoesa (#381) diff --git a/project-bin/exec.sh b/project-bin/exec.sh index a4c85a76..fa10aa12 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -365,7 +365,7 @@ fi change_lane() { case "$CHANGE_CLASS" in ui-only) echo "re-run the journeys of every page the change reaches (embedders of a changed snippet/layout) once + a LOOK. No .test.mdl; reviewers read the artifacts, they do not re-run" ;; - logic) echo "re-run only the tests whose subject this change touched (changed microflows + their SHOW CALLERS, changed pages' journeys). Full mxcli test at module close, not here" ;; + logic) echo "re-run only the tests whose subject this change touched (changed microflows + their SHOW CALLERS, changed pages' journeys). The full suite runs at deploy-ready, per PROJECT.md ## Test strategy" ;; esac } [ -n "$CR_ID" ] && echo "→ Change request: $CR_ID (PROJECT.md '## Change log'; cite this run's commit in its Closed-by)" diff --git a/skills/checkpoints/checkpoint-build.md b/skills/checkpoints/checkpoint-build.md index 11458620..91baa42d 100644 --- a/skills/checkpoints/checkpoint-build.md +++ b/skills/checkpoints/checkpoint-build.md @@ -103,6 +103,27 @@ all scripts upfront. See `brd-to-build-plan.md` ("The build plan contains no MDL `test-plan`, `acceptance-criteria`), or `Optional artifacts: none for Stage 5` as a Stage-4 decision. +**Test strategy (in the same Offer, not a separate question).** Show the defaults and let the +developer change any cell in the same answer: + +> "Testing during the build, unless you want otherwise: +> +> | Moment | Unit tests | Journeys | +> |---|---|---| +> | Every change | affected | affected | +> | Module close | module | module | +> | Deploy-ready | full | full | +> +> `affected` = the tests and journeys of what the change touched and its callers +> (`bin/affected-tests.sh`); `module` = the module's own; `full` = everything; `none` = skip. +> Change a cell, or say 'defaults'." + +**Record as:** `PROJECT.md` → `## Test strategy`, the table plus a `Status` column +(`CONFIRMED`, or `ASSUMED` in auto mode). `bin/affected-tests.sh --moment change|module|deploy` +reads it; with no section it uses these defaults and says so. Unit tests stay subject to the +intake default (`Unit tests default:` in PROJECT.md, `testing-shape.md` §2): `affected` selects +from the tests that exist, it does not ask for new ones. + ## Open Question > "Is there a deadline, a demo milestone, or a specific feature that must be working first? diff --git a/skills/conversion-runbook.md b/skills/conversion-runbook.md index 59d7e8c6..f5800c02 100644 --- a/skills/conversion-runbook.md +++ b/skills/conversion-runbook.md @@ -654,8 +654,8 @@ nothing about the seams. **Read `testing-shape.md` before writing or judging any test here.** It defines what testing a module means, so it is the same thing every time, and it carries the **false-green register**: the -confirmed ways a test reports green while the feature is broken. Its §3 check moments name what the -release gate owes beyond module close. Alongside it, `measured-claims.md` +confirmed ways a test reports green while the feature is broken. Its §3 check moments name what +deploy-ready owes beyond module close. Alongside it, `measured-claims.md` holds the rule that gives evidence its force — *a claim not measured in that register may not be cited as evidence.* diff --git a/skills/existing-app-change.md b/skills/existing-app-change.md index 3ad624be..944c75fd 100644 --- a/skills/existing-app-change.md +++ b/skills/existing-app-change.md @@ -196,6 +196,9 @@ baseline from Ground Rule 1, re-run, and by nothing else. A module gate that is green on the new work and was never run against the baseline has answered half the question and reported it as the whole. Re-run the baseline at every module gate, not once at the end — a regression found three modules later costs the same to fix and much more to find. +At a module gate that means the baseline journeys of the modules the change touched +(`bin/affected-tests.sh --moment module`); the whole baseline runs at deploy-ready. A project that wants +the whole baseline at every module gate says so in PROJECT.md `## Test strategy`. --- @@ -297,5 +300,5 @@ it is the only entry mode where that is true. - A build plan over the change. - When the work arrives as a backlog: a `## Rounds` table in `PROJECT.md`, one row per round, its state current. - When an edit cannot go through mxcli: `docs/manual-steps.md` from `templates/manual-steps.md`, every step with its Check and a ticked Done line naming its evidence. -- **A regression baseline that was green before the change and is green after it** — stated as N journeys over M modules, re-run at every module gate. Without this the rest is unverified. +- **A regression baseline that was green before the change and is green after it**: stated as N journeys over M modules, re-run per module gate over the modules it touched, and whole at deploy-ready. Without this the rest is unverified. - Anything reusable learned about the app → `skills/learned-*.md`; process learnings → `process/process-learnings.md`. diff --git a/skills/iterative-build-loop.md b/skills/iterative-build-loop.md index 01e19b05..3163f07f 100644 --- a/skills/iterative-build-loop.md +++ b/skills/iterative-build-loop.md @@ -618,9 +618,9 @@ step. **Verification is scoped — once per gate, not once per script.** Gate: SYNTAX and Gate: BUILD run inside every `exec.sh`, and they are the only per-script checks. The test passes (happy path, journeys, LOOK) run **once per gate round** over what that round changed, and `bin/gate-check.sh` runs **once per gate round**, -not after every exec. What re-runs after a change, by whom, and when the full suite runs (module -close only, never batch close) is `testing-shape.md` §3 — the single source; `exec.sh` prints the -run's lane (`ui-only` / `logic`). Repeating a pass the artifacts already prove is the over-testing +not after every exec. What re-runs after a change, by whom, and when the full suite runs (deploy-ready +by default, per PROJECT.md `## Test strategy`; never batch close) is `testing-shape.md` §3, the single +source; `exec.sh` prints the run's lane (`ui-only` / `logic`) and `bin/affected-tests.sh` lists its tests. Repeating a pass the artifacts already prove is the over-testing toolkit issue #274 measured: UI-only changes re-proven 5–10×. > **Older ordinals, for anyone arriving from a document that still cites them:** Gate 2 = Gate: diff --git a/skills/testing-shape.md b/skills/testing-shape.md index 1ba1a7d8..640df6f0 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -134,7 +134,7 @@ definition of done (`iterative-build-loop.md`). Without this anchor someone decl > hand (journey report, screenshot, diff) does not?* If you cannot name what it would show, do not run it. **The subject set.** A re-run covers the tests whose *subject* the exec'd script(s) changed — the -script is the change list; `bin/exec.sh` tags each run `ui-only` or `logic` and prints its lane: +script is the change list; `bin/exec.sh` tags its lane and `bin/affected-tests.sh