[field 2026-10-17] exec.sh loses mxbuild evidence, keeps the write on a failed gate, and a stale project copy is not flagged #442

Field report from a field migration project (Next.js + PostgreSQL to Mendix 11.15, 12 modules), Stage 5. Related: #424, #236.

What we were doing

Stage 5 of the conversion, a parallel build. The lead session replayed numbered MDL scripts on the real model one at a time with bash bin/exec.sh <script>.mdl (snapshot, mxcli exec, then the gate: a full mxbuild --target=deploy, then the lint gate, then one BUILD-LOG row). In the first phases a single agent authored scripts. From about 22:57 local, three phases were authored in parallel on private copies of the app while the lead kept replaying on the real model. Throughout, an app was served with ./mxcli run --local --watch -p App.mpr for the look loop. Other Mendix projects also built on the same machine (14 cores, load average up to 68.75).

Commands in use (exact shape):

./mxcli run --local --watch -p App.mpr
bash bin/exec.sh mdlsource/<NN-script-name>.mdl [--commit-first]
mxbuild --java-home=<jdk> --write-errors=<file> --target=deploy App.mpr   # run by _common.sh as the gate
bash bin/verify-model.sh                                                    # standalone model verify that writes the model stamp

Evidence base: docs/BUILD-LOG.md had 115 data rows at the time of writing (exec ledger, local time +0800). Row counts by text: 57 LINT ROSE, 19 gate could not run, 10 UNVERIFIED, 7 --commit-first: git commit refused, 5 uncommitted model changes, 3 mxcli check failed pre-exec.

Log excerpt (BUILD-LOG rows, verbatim)

| 2026-10-16T19:18:27+0800 | `06-domain-ModuleA.mdl` | fail | ❌ gate could not run | mxbuild exit 1 · change: logic · approval: auto |
| 2026-10-16T19:18:28+0800 | `07-domain-ModuleB.mdl` | not-run | 🚫 blocked | uncommitted model changes - refusing exec to prevent snapshot regression |
| 2026-10-16T19:19:38+0800 | `06-domain-ModuleA.mdl` | not-run | 🚫 blocked | --commit-first: git commit refused |
| 2026-10-16T19:19:39+0800 | `07-domain-ModuleB.mdl` | not-run | 🚫 blocked | uncommitted model changes - refusing exec to prevent snapshot regression |

exec.sh console on the failed gate:

Snapshot preserved. Open in SP to verify manually before proceeding.

Mechanism: bin/model-stamp.sh writes a PASS stamp only after a clean gate and clears it otherwise. A "gate could not run" keeps the write (exec.sh around line 1707) and so does UNVERIFIED (around line 1559). The next --commit-first (around lines 676 to 716) hits the pre-commit hook, which refuses a model commit with no stamp unless the last BUILD-LOG row is | pass | and under 4 h old. The commit is refused, a not-run row is written, the model stays uncommitted, and the retry hits the uncommitted-model guard (around line 733). Of 15 blocked rows, 12 follow this chain (7 --commit-first: git commit refused, 5 uncommitted model changes). Retry latency from block to the next applied row of the same script totals about 29 minutes.

Minimal repro: (1) run an exec while mxbuild exits non-zero with no errors file (for example kill mxbuild); (2) run a second valid script with --commit-first. Expected: the second run proceeds, or exec.sh prints the exact recovery command. Actual: commit refused, nothing runs, and the retry is refused again.

Symptom

Two linked defects that amplify every "gate could not run" row.

1. exec.sh cuts mxbuild output with head -20 and does not save it:

echo "$MXBUILD_OUT" | grep -v ... | head -20

head keeps the start and loses the failure tail. _common.sh writes mxbuild output to a mktemp file and deletes it. The detail text of one CE1613 row was lost this way.

2. The project's bin/exec.sh was stale against the toolkit's project-bin/exec.sh. It lacked the fix(exec): roll back the model write when mxbuild crashes #424 rollback on mxbuild crash, so exit 1 runs printed:

Snapshot preserved. Open in SP to verify manually before proceeding.

and kept their write. The next exec was then refused:

uncommitted model changes - refusing exec to prevent snapshot regression

sync-project did not flag the drift.

Repro

1. Use a project whose bin/exec.sh predates fix(exec): roll back the model write when mxbuild crashes #424.
2. Make mxbuild exit 1 without model errors (see the watcher race issue).
3. Observe the kept write and the blocked next exec. Run sync-project.sh and observe no drift report.

Expected vs Actual

• Expected: full mxbuild output saved, tail visible; a crashed gate rolls back; sync-project reports a stale project copy of exec.sh and _common.sh.
• Actual: output truncated and discarded, write kept, drift unreported.

Frequency

12 of 15 blocked rows (7 commit refused, 5 uncommitted model) follow a kept unverified write. Also every mxbuild exit 1 row (17 of 30 failure events in the deep-dive slice), plus one CE1613 row whose detail was lost.

Cost

About 20 minutes in blocked reruns for the lost evidence and stale copy, and about 29 minutes of blocked-retry latency for the stamp cascade.

Proposed fix

• Save full mxbuild output to .mpr-snapshots/last-mxbuild-out.txt and print the tail, not the head.
• Have sync-project.sh (and session-check.sh) compare project bin/exec.sh and bin/_common.sh against the toolkit copies and warn loudly on drift, naming the missing fix.
• On commit refusal after an unstamped kept write, have exec.sh run verify-model.sh once (a clean standalone mxbuild stamps) and retry the commit, or print the exact recovery command. Add the refusal texts to the table in skills/iterative-build-loop.md.
• Backport the fix(exec): roll back the model write when mxbuild crashes #424 rollback to stale copies on sync.

Files

• bin/exec.sh (the head -20 line near the mxbuild output print)
• bin/_common.sh (output tempfile handling around the mxbuild call)
• project-bin/exec.sh
• bin/sync-project.sh
• bin/session-check.sh
• bin/model-stamp.sh, pre-commit hook
• skills/iterative-build-loop.md