From d2edf0b0ebf9cbb58254119265744779a9962a41 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 15:50:05 +0000 Subject: [PATCH 1/7] =?UTF-8?q?feat(exec):=20opt-in=20--on-copy=20?= =?UTF-8?q?=E2=80=94=20exec=20and=20gate=20on=20a=20scratch=20copy,=20prom?= =?UTF-8?q?ote=20by=20rename=20(#350)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EXEC_ON_COPY=1 or --on-copy runs mxcli exec and the mxbuild gate on a copy of the model tree under the existing exec lock, and writes the live model only by a rename-based promotion after a verified pass. Promotion refuses while Studio Pro holds the model, when the live model changed since the copy, or across filesystems; a part-way swap renames back or restores the snapshot. Any other outcome leaves the live model untouched and keeps the copy with a resume hint. The default path is unchanged. Copy routine shared with test-scratch.sh via _common.sh (mxtk_copy_model_tree). Fixture tests/wave2/test-exec-on-copy.sh (108 checks, both layouts) and bench tests/wave2/bench-exec-on-copy.sh. Field evidence: none (stub-only). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 1 + project-bin/_common.sh | 12 ++ project-bin/exec.sh | 310 ++++++++++++++++++++++++++++-- project-bin/test-scratch.sh | 3 +- skills/iterative-build-loop.md | 2 + tests/wave2/bench-exec-on-copy.sh | 179 +++++++++++++++++ tests/wave2/test-exec-on-copy.sh | 236 +++++++++++++++++++++++ 7 files changed, 722 insertions(+), 21 deletions(-) create mode 100755 tests/wave2/bench-exec-on-copy.sh create mode 100755 tests/wave2/test-exec-on-copy.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index ff60ea83..d1f31fb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- new(exec): **`exec.sh --on-copy` (or `EXEC_ON_COPY=1`), opt-in: the script and the mxbuild gate run on a scratch copy, and the live model is written only by a rename-based promotion after a verified pass** ([#350](https://github.com/MendixMau/mxcli-project-toolkit/issues/350)). By default the live model holds an unverified write for the length of the gate (22-65 s per apply in the field), and keeps it when exec.sh is killed. Promotion is refused while Studio Pro holds the model (even under `FORCE_EXEC`), when the live model changed since the copy (cksum manifest), or when the copy is on another filesystem. A swap that fails part-way renames back, else restores the snapshot. On any other outcome the live model is untouched and the copy is kept under `.mpr-snapshots/.on-copy/` with a resume hint (newest 3 kept). Files exec writes outside the model (javasource/) are promoted with it. The default path is unchanged. The copy routine moved from `test-scratch.sh` into `_common.sh` (`mxtk_copy_model_tree`). Fixture `tests/wave2/test-exec-on-copy.sh` (108 checks, both layouts; 71 fail on master). Bench `tests/wave2/bench-exec-on-copy.sh` (stub mxbuild 2 s, an input): with exec.sh killed mid-gate the live model stayed byte-identical in 0/10 default runs and 10/10 on-copy runs at 500, 5,000 and 20,000 units. The broken window fell from about 2.1 s (clean) and 5.5-25.4 s (failed gate) to a 15-42 ms swap, at a cost of +1.4 s, +3.7 s and +10.9 s wall time per exec. Field evidence: none yet (stub-only); a field run with Studio Pro closed and then open is owed before it can become the default. Statement-level resume from the kept copy is a follow-up. — external review (#350), MendixMau - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/project-bin/_common.sh b/project-bin/_common.sh index 48113307..27bbf05c 100755 --- a/project-bin/_common.sh +++ b/project-bin/_common.sh @@ -642,6 +642,18 @@ mxtk_mxbuild_why() { "$_py" -c "import json;d=json.load(open('$(native_path "$1")'));print('; '.join((e.get('message','') if isinstance(e,dict) else str(e)) for e in d.get('errors',[]) or [])[:300])" 2>/dev/null || true } +# mxtk_copy_model_tree — copy the model tree (the .mpr, mprcontents/ and +# everything mxbuild reads beside them: widgets/, javasource/, theme/, userlib/ ...) into an +# existing . Leaves out .git, .mpr-snapshots, deployment, .mxcli (the LIVE app's +# run --local record — a copy carrying it points app-restart.sh at the live app) and node_modules. +# One routine for both scratch-copy users: bin/test-scratch.sh (mxcli test, #278) and +# bin/exec.sh --on-copy (#350). tar preserves mtimes, which --on-copy relies on to tell the +# files `mxcli exec` wrote from the files it copied. Returns tar's status. +mxtk_copy_model_tree() { + tar -C "$1" --exclude=./.git --exclude=./.mpr-snapshots --exclude=./deployment \ + --exclude=./.mxcli --exclude=./node_modules -cf - . | tar -C "$2" -xf - +} + # project_name — the .mpr basename without extension, for user-facing messages. project_name() { local mpr diff --git a/project-bin/exec.sh b/project-bin/exec.sh index b85dde08..f57c2859 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -17,7 +17,25 @@ # ./bin/exec.sh --watch-compat (advisory: can `run --local --watch` apply # this, or does the app need a restart; writes nothing) # ./bin/exec.sh --cr CR-07 (any mode: tag the run with its change request) -# Leading flags (--cr, --commit-first, then --patch) may come in any order. +# ./bin/exec.sh --on-copy (opt-in, or EXEC_ON_COPY=1: write a scratch +# copy, promote it only on a clean gate) +# Leading flags (--cr, --commit-first, --on-copy, then --patch) may come in any order. +# +# --on-copy (opt-in, 2026-10-14, #350): `mxcli exec` and the mxbuild gate run on a scratch COPY +# of the model (.mpr-snapshots/.on-copy//, the tree mxtk_copy_model_tree copies), and the +# live model is written only by a promotion after a verified pass with exit 0. Without it, the +# live model holds the script's changes from the first statement until the gate ends or the +# snapshot is restored: 22-65 s per apply in the field, and forever if exec.sh is killed meanwhile. +# Promotion refuses (live untouched, copy kept) when Studio Pro holds the model, even under +# FORCE_EXEC, or when the live .mpr/mprcontents changed since the copy was taken (cksum manifest), +# or when the copy is not on the model's filesystem (a rename would become a copy). The swap is +# renames only: .mpr is replaced by one rename (atomic); mprcontents/ by two (live -> aside, new -> +# live), so for the microseconds between them there is no mprcontents/ (rename(2) cannot exchange +# two directories portably). Files `mxcli exec` wrote outside the model (javasource/ for a Java +# action) are promoted the same way, one rename each; a file it DELETED there is not mirrored +# (follow-up). A swap that fails half-way renames back, else restores the pre-exec snapshot. On any +# other outcome the live model is never written and the copy is kept with a resume hint; the +# newest MXTK_ON_COPY_KEEP (default 3) kept copies survive. Not with --patch (follow-up). # # --cr (2026-10-13): names the PROJECT.md "## Change log" row this run applies # (skills/change-intake.md). It lands in the BUILD-LOG detail cell as "cr: CR-07", the same way @@ -171,6 +189,7 @@ mxtk_change_class() { # $1 = MDL script; prints "\t" # `[--commit-first] [--cr CR-NN]`; in that order commit-first committed the model and the run # went on with "--cr" as its script path. CR_ID=""; COMMIT_FIRST="${MXTK_COMMIT_FIRST:-0}"; PATCH_MODE=0; ADVISORY="" +ON_COPY="${EXEC_ON_COPY:-0}"; [ "$ON_COPY" = 1 ] || ON_COPY=0; ON_COPY_FLAG=0 while [ $# -gt 0 ]; do case "$1" in --cr) @@ -185,6 +204,7 @@ while [ $# -gt 0 ]; do fi shift 2 ;; --commit-first) COMMIT_FIRST=1; shift ;; + --on-copy) ON_COPY=1; ON_COPY_FLAG=1; shift ;; --patch) PATCH_MODE=1; shift; break ;; --change-class|--watch-compat) ADVISORY="$1"; shift; break ;; *) break ;; @@ -193,7 +213,7 @@ done case "${1:-}" in --*) echo "✗ Unknown option or misplaced flag '$1' — nothing was committed, snapshotted or executed." >&2 - echo " Usage: ./bin/exec.sh [--commit-first] [--cr CR-NN] " >&2 + echo " Usage: ./bin/exec.sh [--commit-first] [--cr CR-NN] [--on-copy] " >&2 echo " ./bin/exec.sh [--commit-first] [--cr CR-NN] --patch