feat(exec): opt-in --on-copy — exec and gate on a scratch copy, promote by rename (#350)

[MendixMau]

Direct lane

What changed and why (one paragraph):
Toward #350. Today exec.sh writes the live model first, then runs mxbuild, then restores on a failure. For the whole gate the live model holds an unverified write, and if exec.sh is killed it keeps that write. This PR adds an opt-in --on-copy flag (or EXEC_ON_COPY=1). Under the existing exec lock, the model tree is copied to .mpr-snapshots/.on-copy/<stamp>/model. The copy routine is now mxtk_copy_model_tree in _common.sh, shared with test-scratch.sh:

• Clone first. Each entry is cloned where the filesystem can (macOS APFS cp -c, Linux reflink), probed read-only on the .mpr, else tarred. MXTK_COPY_MODE=tar|clone overrides the probe.
• No build caches. node_modules, .cache, .mendix-cache and .gradle (MXTK_COPY_SKIP) are never copied, and change detection prunes exactly the same list. Because .mendix-cache is not copied, a gate that needs it may re-download it into the copy; the macOS re-time should confirm what that costs.
• Single-tree checkout: an allow-list (the .mpr and MXTK_APP_DIRS), never sources/, analysis/, docs/ or binaries.
• Two-tree app/: copied whole, minus the caches.

mxcli exec and the mxbuild gate then run on the copy. Only a verified pass of a fully applied script is promoted. Promotion is refused, with the live model left untouched, in these cases:

• Studio Pro holds the model. This is the same sp_holds_model test as the opening guard, which runs before the uncommitted-model guard on every path. With --on-copy or --recover-promote, FORCE_EXEC is refused up front while Studio Pro is open, and a 🚫 blocked (Studio Pro holds the model) BUILD-LOG row is written.
• The live .mpr, its -wal or mprcontents/ changed since the copy was taken. A cksum manifest is taken before the snapshot and checked again right before the swap.
• A file beside the model that the promotion would replace changed live since the copy was taken.
• mxcli exec wrote outside the model and outside OC_SIDE_DIRS.
• A -wal is not empty, either live or in the copy.
• The copy sits on another filesystem. A hard-link probe checks this, because a cross-filesystem mv would copy.

The swap uses renames only, and the old files are kept beside the new ones until the swap is verified:

• The .mpr is replaced by a single rename(2), which is atomic.
• mprcontents/ needs two renames: live to aside, then new to live. For the microseconds between them there is no mprcontents/. Portable mv cannot exchange two directories, and coreutils --exchange (renameat2) only exists from 9.5; this box has 9.4, so that path is untested and left as a follow-up.
• A unit-count check follows the swap. Only then is the .mxtk-promote.N.committed marker written and the old files cleaned up.
• A swap that fails part-way is undone by the same recovery function as below, which hands every new file back to the kept copy. If that also fails, it restores the pre-exec snapshot.
• Files mxcli exec wrote beside the model under OC_SIDE_DIRS (javasource/, javascriptsource/, theme/, themesource/, resources/, nativemobile/) are found by checksum, one batched cksum with caches pruned, against the pre-copy manifest. So an archive-dated file still counts. They are promoted the same way. <mpr>-wal/-shm/-journal/.lock never count as such files. Files exec deleted are not mirrored; that is a follow-up.

A promotion killed part-way leaves .mxtk-promote.<pid>.* names beside the model. The manifest, written before anything else, also records whether the live model and the copy had mprcontents/. While those names exist, every exec.sh path (the default one included) and test-scratch.sh's copy refuse to run, and the message names one command: ./bin/exec.sh --recover-promote. That command runs mxtk_promote_recover in _common.sh under the exec lock; it refuses while Studio Pro holds the model and skips the STOP-row scan.

• Committed marker present: it only finishes the cleanup of the old files; the live model is the promoted one. If the marker is the only file left, it is removed.
• No marker: it rolls back by rename using the manifest, with every step guarded by what is on disk, so a second run is a no-op. A hard-linked .old that still -efs the live file means "never replaced". New files go back to the kept copy, or into a trash dir that is removed only after every rename succeeded. It never deletes a live file.
• Other leftovers without a manifest: refused untouched (exit 2).

No shell commands are printed. restore-mpr.sh is mentioned only as the last resort. On a clean model the detection costs one glob.

On any other outcome (gate fail or unverified, mxcli exec non-zero) the live model is never written. The copy is kept with its path and a resume hint, and only the newest MXTK_ON_COPY_KEEP (default 3) copies are kept. Applied, lint and score rows wait until after the promotion, so every outcome still writes exactly one BUILD-LOG row. The verification stamp is written exactly as on the normal path. --on-copy --patch is refused; EXEC_ON_COPY=1 with --patch prints a note and runs the patch as before. The default path is unchanged except for the leftover refusal, which can only fire after an interrupted --on-copy promotion. The sp_holds_model refactor is equivalent, and the extra detail suffix and target variable are empty or equal to $MPR when the flag is unset. test-scratch.sh refuses with a sync remedy when an older _common.sh lacks the copy routine.

Impact (five rows, checked against the diff; measurements on macOS 26 arm64, APFS, bash 3.2, a lab project with 549 units, Mendix 11.15):

Row	Answer
Process	Opt-in. No new human step on the default path. New blocking paths: a leftover .mxtk-promote.* beside the model makes every exec.sh run and test-scratch.sh refuse until ./bin/exec.sh --recover-promote, reachable only after an interrupted --on-copy promotion; test-scratch.sh exits 3 when an older _common.sh lacks mxtk_copy_model_tree; under --on-copy and --recover-promote, FORCE_EXEC no longer overrides an open Studio Pro. New stage artifact: up to MXTK_ON_COPY_KEEP (3) kept model copies under .mpr-snapshots/.on-copy/. Without mxbuild the gate is skipped, so nothing is promoted.
Dependencies	Shared files: project-bin/exec.sh, project-bin/_common.sh (new mxtk_copy_model_tree, mxtk_copy_how, mxtk_promote_leftovers, mxtk_promote_recover), project-bin/test-scratch.sh, skills/iterative-build-loop.md. Sync exec.sh and _common.sh together (explicit refusals otherwise). New env knobs: EXEC_ON_COPY, MXTK_COPY_MODE, MXTK_COPY_SKIP, MXTK_APP_DIRS, MXTK_ON_COPY_KEEP. New artifact read: .mxtk-promote.<pid>.manifest, written by the same run before any rename. Needs the copy on the model's filesystem (hard-link probe). Merge conflicts with master are CHANGELOG.md (keep both) and two exec.sh hunks against #374 (straight union). Order: before #311 and #379, which also lock exec.sh.
Context cost	skills/iterative-build-loop.md +1 paragraph, not baseline-routed. bin/render-routing.sh --check: stage 5 69,795 words / 25 files (origin/master) to 69,799 / 25 (this head, which has not merged the newest master; the +4 is the file's own growth in that stage's role views). A review on a merge-tree result reported the baseline unchanged at 75,225 of 80,000.
Behaviour	Situational skill text: an A/B is recommended, NOT RUN.
Runtime	Default path: one glob. --on-copy on the real model on APFS with cp -c (3 runs each, alternating, exec.sh wall seconds): default 144 / 88 / 73 (median 88), on-copy 394 / 233 / 170 (median 233); copy step 80.1 / 45.5 / 32.3 s, swap 53 to 199 ms. The host was under heavy load from other processes (load average 270 falling to 70 during the series), so read all figures as upper bounds. The gate on the copy cost about 65 s more than the copy itself in the last pair (not isolated; the copy has no .mendix-cache).

macOS re-time and Ctrl-C (owed items): a SIGINT sent to the exec process group 4 s after the copy was taken (mid exec and gate) exited 130, left the live .mpr and mprcontents/ checksum-identical, left no .mxtk-promote.* names and no stray process; the scratch copy is kept by design under .mpr-snapshots/.on-copy/ with a resume hint and is pruned to the newest 3. Test fixture on macOS: 275 passed / 8 failed before, 283 / 0 after b65a16a (TMPDIR trailing slash, busy port 8081).

Field run: #390

macOS 26.6.2 arm64, bash 3.2.57, Mendix 11.15.0, mxcli v0.26.0, one real app with 549 units (MPR v2), tested at fff1cd5.

• a) Studio Pro closed: PASS. The copy was built and checked, then promoted with one BUILD-LOG row and exit 0. The swap took 107–138 ms.
• b) Studio Pro holds the model: PASS. With a lock file naming a live process, plain and FORCE_EXEC=1 runs were both refused with exit 1, and the model was unchanged.
• c) Interrupted promotion: only the no-op --recover-promote half was run (twice, both "Nothing to recover"). The Ctrl-C half was NOT RUN.

Findings, now fixed in a463e69:

• The copy dominated: 86.7 s unloaded and 133.8 s loaded, so each on-copy run took 4–7 min. The fixes are cloning, cache exclusion and batched checksums; the bench below shows the effect.
• A Studio Pro refusal wrote no BUILD-LOG row. It now writes one.
• Guard order. In b1, the dirty-model guard fired, but the Studio Pro check was not skipped: the report's own b2 shows Studio Pro had opened a different clone and there was no .mpr.lock beside the field model. The Studio Pro guard already runs before the uncommitted-model guard. A new fixture case (dirty model plus lock, --on-copy) proves the Studio Pro refusal comes first.

Owed before it can become the default: the Ctrl-C half of c on a real model, and a re-time of the copy on macOS APFS with the clone path, which should also show whether the gate re-downloads anything because .mendix-cache was not copied. The cp -c path is not exercised on Linux; the fixture forces MXTK_COPY_MODE=clone, which uses cp -a --reflink=auto here.

Copy cost, before → after (BENCH_APP_TREE=1 BENCH_SIZES=550, 3 clean runs per path, median; 2 s stub mxbuild). Linux ext4, so there is no reflink and the copy is tar in both runs. The model is 550 units plus a realistic app tree: javasource 1,500 files, javascriptsource 100 files plus a 15,000-file node_modules, themesource 400, theme 60, widgets 40 × 1 MB, userlib 60 × 1 MB, vendorlib 150 × 1 MB.

code	copy (ms)	swap (ms)	wall default → on-copy (ms)	killed mid-gate: live identical default / on-copy
before (fff1cd5 + master)	758	14	3818 → 83331 (+79513)	0/2 / 2/2
after (a463e69)	1036	21	4127 → 5876 (+1749)	0/2 / 2/2

Most of the overhead on this box was the per-file cksum loop over the side directories, including the node_modules, run once per file and twice per exec. The macOS 87–134 s was the copy itself, which cloning addresses and which still needs a field re-time.

Before/after measurement, broken window (tests/wave2/bench-exec-on-copy.sh; every number below is computed by the script; measured on d2edf0b, model-only synthetic units). The stub mxbuild duration is an input, set to 2 s here. Real mxbuild measured 22–65 s per apply, per iterative-build-loop.md, so the default path's real broken window is that long or longer. The real exec.sh, snapshot-mpr.sh, restore-mpr.sh and _common.sh were used. Clean-run figures are the median of 3 runs. A poller cksums the live .mpr + mprcontents/ in a loop:

• Default window: from the first poll that sees the write to the end of mxbuild (clean run), or to the first poll that is back at the original (failed gate, restored).
• On-copy window: exec.sh's own timer around the renames.
• "Mixed-state polls": polls that saw neither the original nor the promoted state. Each one is a poll that straddled the swap, because one cksum pass over 20k files takes about 385 ms.
• Failure injection: 10 runs per path per kind. "Gate-fail" means the stub reports a new CE. "Killed mid-gate" means exec.sh is SIGKILLed while mxbuild runs.

units	broken window, default (ms)	broken window, on-copy (ms)	mixed-state polls on-copy	poll period (ms)	copy (ms)	swap (ms)	wall default → on-copy (ms)	gate-fail: live identical default / on-copy	killed mid-gate: live identical default / on-copy
500	2103 clean · 5456 failed gate	42	3 of 331	62	198	42	4816 → 6240 (+1424)	10/10 / 10/10	0/10 / 10/10
5000	2077 clean · 7476 failed gate	15	3 of 267	121	1907	15	6865 → 10589 (+3724)	10/10 / 10/10	0/10 / 10/10
20000	2111 clean · 25404 failed gate	21	3 of 210	385	5153	21	10044 → 20936 (+10892)	10/10 / 10/10	0/10 / 10/10

How to read it:

• The gain: a killed exec leaves the live model intact in 10/10 runs instead of 0/10. The broken window drops from the gate length (2 s stub, 22–65 s real) plus restore time to a 15–42 ms rename.

• The cost: +1.4 s, +3.7 s and +10.9 s wall time per exec at 500, 5,000 and 20,000 units on the original code. The wall-time figures were measured with the poller competing for CPU, so treat them as upper bounds.

• Gate failures: both paths already left the live model identical (10/10).

• No client data anywhere in the diff

• Size cap: 7 files, but over the 400-line cap. 573 non-fixture lines changed against the merge-base: exec.sh +363/−19, _common.sh +169, test-scratch.sh +16/−3, skill +2, CHANGELOG +1. The growth is the review- and field-requested safety and copy-cost work. It could be split into the core path plus a recovery follow-up, but the core path is not safe without the recovery.

• Test tier reached: T1 stub fixtures, plus the T3 field run above (Field runs 2026-10-15: #379 PASS, #384 PASS (sync --upgrade-bin not scoped; copy 87-134 s), #374 PASS (.bak in mprcontents) #390, a and b). test-exec-on-copy.sh passes 283/0 across both layouts. It covers:

  • clean promotion and failed gate;
  • a partial exec;
  • a live model or side file changed during the exec;
  • Studio Pro opened mid-exec;
  • a failed swap renamed back with the copy whole;
  • a kill in each of 8 promotion windows, recovered twice: the live model ends up as the pre-exec or the promoted model, never a mix;
  • a lone .committed marker;
  • --recover-promote under Studio Pro;
  • leftovers without a manifest;
  • an archive-mtime side file;
  • a stray write, and -journal exclusion;
  • FORCE_EXEC with Studio Pro open, now with its BUILD-LOG row;
  • a dirty model with Studio Pro open (the Studio Pro refusal comes first);
  • copy modes clone and tar: caches not copied, mtimes preserved, promotion works, live node_modules untouched;
  • the single-tree allow-list;
  • the env variable, pruning and --patch.

  After merging origin/master: test-exec-lock-watch 84/0, test-exec-change-class 21/0, test-bug07-08 43/0, test-test-scratch 51/0, test-shared-lock-takeover 6/0, test-mdl-scan (with this exec.sh) 52/0.

• New/changed instrument: both layouts covered (.mpr at the root and under app/). Platforms: plain bash 3.2 constructs, no flock, tar names passed as arguments, a ms timer that falls back to whole seconds on BSD date, and cp -c gated on mxtk_platform = macos and probed before use. Field-run on macOS at fff1cd5; the clone path is not yet field-run. The bench needs GNU date.

• CHANGELOG.md line appended, credited, and updated with the Field runs 2026-10-15: #379 PASS, #384 PASS (sync --upgrade-bin not scoped; copy 87-134 s), #374 PASS (.bak in mprcontents) #390 results.

• Docs: one paragraph in skills/iterative-build-loop.md under the gate invariants (+199 words in that file, including the .mendix-cache note). The baseline word count is unchanged by this PR, at 73,522 / 80,000 after the master merge, because that file is not in the baseline.

Follow-ups (not in this PR):

• statement-level resume from the kept copy;
• --on-copy for --patch;
• mirroring files exec deleted outside the model;
• an atomic directory exchange where mv --exchange / renameat2 exists;
• adding /.mpr-snapshots/ to init-project.sh's scaffolded .gitignore;
• a scoped sync-project.sh install (Field runs 2026-10-15: #379 PASS, #384 PASS (sync --upgrade-bin not scoped; copy 87-134 s), #374 PASS (.bak in mprcontents) #390 finding 1);
• the Ctrl-C field half and a macOS clone re-time.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw