Skip to content

new(testing): affected-tests.sh and exec.sh --subjects, advisory, selection gaps closed (409b) - #416

Merged
MendixMau merged 4 commits into
masterfrom
feat/affected-tests
Oct 16, 2026
Merged

new(testing): affected-tests.sh and exec.sh --subjects, advisory, selection gaps closed (409b)#416
MendixMau merged 4 commits into
masterfrom
feat/affected-tests

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Direct lane

What changed and why (one paragraph):
Split out of draft #409 (second of three; #413 is the first, the held docs PR is the third). Adds project-bin/affected-tests.sh <script.mdl>, which prints the unit tests and journeys a change owes, and exec.sh --subjects <script.mdl>, which lists the elements a script writes (one kind Module.Name line N row each). testing-shape.md §3 described this subject set in prose only. The script takes the subjects from --subjects, gets their transitive callers (pages included on mxcli v0.26.0) from ONE batched SHOW CALLERS OF ... TRANSITIVE call, and lists the .test.mdl and journeys/<Module>.journey.json files owed at the moment's scope, read from a PROJECT.md ## Test strategy row when there is one (defaults when absent, and it says so). Advisory only: exit 0, never runs a test, no doc says it is binding. exec.sh prints one affected tests: hint line per MDL run. The three selection gaps the review of #409 found are fixed here:

  1. journeys unmapped N of M: counts the journey files that name-based selection can never reach (not <Module>.journey.json, no qualified Module.Name inside), so an empty journey selection is never read as "nothing owed".
  2. A journey file that names a transitive caller (a page at any depth) is selected, not only one that names a subject.
  3. An entity, association, enumeration, access or module subject prints a data NOT MEASURED row: cross-module use is not measured, only the owning module is selected.
    Caller depth stays configurable, default 1: --depth N, or MXTK_AFFECTED_DEPTH for the default. Depth 2 is decided later from the offline selection survey, not here.

Impact

Row Answer
Process none. Advisory: exit 0 (2 on usage), no new human step, gate, blocking path or stage artifact. It reads PROJECT.md ## Test strategy if present, but nothing writes that section until the held docs PR.
Dependencies Shared files: project-bin/exec.sh (the --subjects flag and hint line only) and bin/lib/install-manifest.sh (new project-bin entry; installed by sync). Independent of #413: that PR touches other parts of exec.sh, so the two merge in either order and only the CHANGELOG line needs a trivial rebase for whichever lands second. Optional artifact read: PROJECT.md ## Test strategy, producer is the CAC-5 Offer in the held docs PR; absent means defaults, stated in the output. Needs mxcli with SHOW CALLERS ... TRANSITIVE (v0.26.0 verified); without it the output says callers NOT MEASURED.
Context cost none: no routed or always-loaded file touched
Behaviour n/a: scripts only
Runtime Advisory and optional, not on the exec path beyond one echo. Measured on a scratch copy of a five-app lab project: first call 19 s (mxcli catalog build), later calls 3 s.

Field evidence: a five-app lab project (Mendix 11.15, mxcli v0.26.0), scratch copy, three real change scripts: 2 to 55 subjects, 8 to 17 transitive callers, 0 to 1 unit test files and 0 to 1 of 20 journey files selected. 18 of its 20 journey files are named by app and role and 12 of 20 hold no qualified name, which is gap 1 and why the unmapped count exists. Golden callers output captured from mxcli v0.26.0 on a scratch copy of an empty Studio Pro 11.12 app (tests/wave2/fixtures/affected-tests/CAPTURE.txt).

  • No client data anywhere in the diff
  • Size cap: over. 5 files and 541 changed lines excluding fixtures and CHANGELOG (affected-tests.sh 242, exec.sh 44, manifest 2, tests 253). About half is test code, and the script and the tests that prove it cannot ship apart; the split that could be made (script without tests) would put untested code on master.
  • Test tier: T1, scoped fixtures. tests/wave2/test-affected-tests.sh 54/0 (13 pass and 41 fail with no script, as on master). Against process(testing): per-project test strategy; affected tests per change, full suite at deploy-ready #409's original script 48 pass and 6 fail: exactly the six gap checks (caller journey, unmapped count, env depth, data row for entity and enumeration, mixed script). tests/wave2/test-exec-subjects.sh 6/0 (3 pass and 3 fail on master's exec.sh). Existing: test-exec-change-class.sh 21/0, test-manifest-clean-checkout.sh 1/0. Only macOS was run; Git Bash was not.
  • Golden input captured, not hand-written. Both layouts: find_mpr supports root and app/.
  • CHANGELOG.md line, credited
  • Categorisation: kind new · lane Direct · size L (note above) · regression risk low (advisory) · toolkit change, not an upstream mxcli bug

Known and not fixed here: in a multi-app folder the script scans every app's tests and journeys and needs MPR_FILE; module names can repeat across apps, so the selection over-reaches. That waits on the model key work.

🤖 Generated with Claude Code

@MendixMau @claude
new(testing): affected-tests.sh and exec.sh --subjects, advisory, wit…
b91faaf 
…h the three selection gaps closed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 16, 2026
Merged
Closed
MendixMau and others added 3 commits October 16, 2026 11:00
@MendixMau
MendixMau merged commit b46db5a into master Oct 16, 2026
2 checks passed
MendixMau added a commit that referenced this pull request Oct 16, 2026
@MendixMau
fix(exec): per-step seconds in the run summary and BUILD-LOG, gate se…
fb58cee 
…conds kept on a failed gate (#413)

Independent review: APPROVE.

# Review #413 (409a): exec.sh per-step seconds, gate seconds kept on a failed gate

Reviewed 2026-10-16 at head `f118bb0f` (base `1ed7c99a`), detached worktree `/tmp/rv-413`, read only. No GitHub comment.

## What it does

`step_done <name>` appends `<name> <s>s` since the last mark. Eight marks: guards, check, snapshot, baseline, exec, gate, lint, score. The run summary prints `steps: ...` on success. `log_build` appends `· steps: ...` as the last item of the BUILD-LOG detail cell. The `gate` mark sits right after the mxbuild call, before the pass and fail branches. That covers big-topics T1 finding (a).

## Checks

| Check | Result |
|---|---|
| Test fails on master | `test-exec-step-times.sh`: branch 7/0, master's exec.sh 4/3 (summary line, BUILD-LOG cell, failed-gate cell). Confirmed. |
| Failed-gate timing | The failed `--patch` row carries `... exec Ns gate Ns`. MDL gate-fail rows (exec.sh 1700 to 1716) and PARTIAL rows sit after the same mark, so they get it too. |
| Early-exit paths | Blocked rows before the guards (scan, commit-first, dirty model) have no cell, because STEP_TIMES is empty. The pre-exec check failure logs `guards Ns`. Ctrl-C writes no BUILD-LOG row on master either (the EXIT trap only releases the lock), so nothing regresses. |
| Gap | `❌ patch failed` / `❌ patch refused` rows (exec.sh 1429, 1517) log before `step_done exec`, so they lose the patch run seconds. Non-blocking. |
| Readers | `status.sh` reads columns 3 and 4 only. `done-drift-check.sh` greps the script name. `metrics-ledger.sh` takes args from exec.sh and does not parse the detail. The new text is last in column 6 and has no `\|`. No reader breaks. |
| Bash 3.2 | Test 7/0 with `/bin/bash` 3.2.57 first on PATH. |
| Neighbours | `test-exec-change-class.sh` 21/0, `test-exec-lock-watch.sh` 84/0. `test-exec-hash-verify.sh` 42/1 (case K). The same case fails on master, so it predates this PR. |
| Conflict with #416 | `git merge-tree`: exec.sh auto-merges, CHANGELOG.md conflicts. The merged exec.sh passes both PRs' tests (7/0 and 6/0). |
| Hygiene | No client names. CHANGELOG line present. 17 code + 94 test lines, under the size cap. CI authoring-audit green. |

## Impact block (RELEASE-PROCESS 4a)

| Row | Verdict |
|---|---|
| Process | Confirmed: none. |
| Dependencies | Confirmed. exec.sh is shared. The detail-cell readers were checked above. |
| Context cost | Confirmed: none. |
| Behaviour | Confirmed: n/a, scripts only. |
| Runtime | **Corrected.** The PR calls the cost "unmeasurable". Measured here: one `date +%s` takes 22.8 ms under bash 3.2 on this Mac (90-call loop). Nine calls add about 0.2 s per exec, 0.5 to 0.7% of a 28 to 41 s exec. Small but real, and it runs on every exec in every project. Optional: use bash's built-in `$SECONDS` (works in 3.2, no fork) to make it about 0. |

5 of 5 rows confirmed or corrected.

## Non-blocking notes

1. Add `step_done exec` before the two patch-failure `log_build` calls (exec.sh 1429, 1517), so a failed patch logs how long it ran.
2. Put the measured 0.2 s in the Runtime row, or switch to `$SECONDS` and state 0.
3. On failure, the steps appear only in BUILD-LOG, not on the console. That is fine for E3 but worth one sentence in the CHANGELOG line, which says "run summary".

## Verdict

**APPROVE.** Merge first, before #416. The second PR to land needs a CHANGELOG-only rebase.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MendixMau