From 4ab387198a734f30b6331c0af50e8387a0efc699 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Sat, 17 Oct 2026 00:38:08 +0800 Subject: [PATCH] fix(exec): pause the project's watcher during the gate and retry once on an infra exit The gate's mxbuild and a live `mxcli run --local --watch` both write deployment/. In a field migration project (12 modules) 17 of 27 failed gates were "gate could not run" (mxbuild exit 1, no CE code), against 0 of 22 in watcher-free copies. The watcher of this project is stopped for the gate and resumed after (and in the EXIT trap). An exit 1 or 3 with no CE code and no errors file is retried once and logged as "retry after infra exit". A final failure still takes the #424 rollback. Refs #441, #442, #171, #285, #424 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + project-bin/_common.sh | 84 ++++++++++++++++++ project-bin/exec.sh | 13 +-- tests/wave2/test-exec-gate-retry.sh | 127 ++++++++++++++++++++++++++++ 4 files changed, 220 insertions(+), 5 deletions(-) create mode 100644 tests/wave2/test-exec-gate-retry.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 27fa96ab..06bee061 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- fix(exec): **the gate pauses this project's `mxcli run --local --watch` while mxbuild runs and retries once on an infra exit.** The gate's mxbuild and a live watcher both write `deployment/`; in a field migration project (12 modules) 17 of 27 failed gates were "gate could not run" (mxbuild exit 1, no CE code), against 0 of 22 in watcher-free copies. The watcher is stopped (SIGSTOP) for the gate and resumed (SIGCONT, also in the EXIT trap); an exit 1 or 3 with no CE code and no errors file is retried once and the BUILD-LOG row says `retry after infra exit`; a final failure still takes the #424 rollback. Test: `tests/wave2/test-exec-gate-retry.sh` (Refs #441, #442, #171, #285, #424) — field migration project - learn(unit-test): the unit-test skill names the per-flow `Unit flows:` list next to `Unit modules:`, so whether a flow gets a test reads the same in both places (follow-up to #435) — MendixMau - learn(load-test): **new on-demand skill for load testing a page without correlating Mendix's client protocol by hand: Playwright records the real `/xas/` calls, a probe maps each opaque `runtimeOperation` to its microflow through Jaeger, and the chosen call is replayed with the session's own cookie and CSRF token.** Ten field traps (HTTP 200 on a failed action, caught REST errors under a green microflow span, the wrong call hammered, a recording that still held the previous page, a tracing check that failed open, a harness that throttled itself, harness vs app limits, requests vs users, sampled trace counts, empty data) and a four-line report that says what the run does not prove. Routed ondemand for lead, test and review; no instrument ships yet and the roadmap's "no load testing" line is reworded. Field evidence: a warehouse demo app, Mendix 11, 16 runs 2026-09-09 to 2026-10-08, bursts of 1000 and 2000 concurrent calls on one session — MendixMau - new(process-map): **Process map: one swimlane map per business process, each step on its wireframe once screens exist, every open question and ASSUMED decision pinned to the step it is about; drawn twice, a draft at Stage 2 close without screens (pass 1, shown at CAC-3) and the full map at Stage 3 with screens (pass 2, walked at the Stage 3 gate close-out).** `skills/process-map.md` (when, model, anchoring, build, verify by looking) and `project-bin/process-map.js` (`validate`, `anchor`, `build`; Node stdlib, deterministic) with its template. Named "map" because "process" in the progress report means the toolkit's own stages, while this shows the app's business process. With no wireframes, a step may name a screen that is not drawn yet or have none, and a person step stays a person step; once a wireframe exists `validate` fails a step page with no wireframe, a use case on no process and not standalone, a decision with one outcome, two steps in one cell. `anchor` places items mechanically (named microflow, listed or named use case, named screen, overlapping source lines) and keeps judged placements by fingerprint, so a reworded question is unplaced again instead of silently wrong; global buckets are allowed and counted. Wired into the Stage 3 Agent produces and Surface rows, a `process-map` opt-in manifest row (an existing model counts as opting in), the `swimlanes` opt-in row (the model discharges it), CAC-3 and CAC-4, and a check_stage_3 FAIL when the map is older than its model, its anchors or a wireframe. `tests/wave2/test-process-map.sh` on a synthetic order approval fixture with its own committed BRD, covering both passes. Field evidence: a requirements-driven rebuild POC, 2026-10-16, where the reviewer could not tell where the sixty questions fit; the map placed 102 items with 0 unplaced. — MendixMau diff --git a/project-bin/_common.sh b/project-bin/_common.sh index dee18cee..67e92889 100755 --- a/project-bin/_common.sh +++ b/project-bin/_common.sh @@ -569,6 +569,90 @@ find_java_exe() { esac } +# --------------------------------------------------------------------------- +# Watcher pause and infra retry around the exec gate's mxbuild. +# +# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live +# `mxcli run --local --watch` on the SAME project both write deployment/. One of the two dies +# part-way ("Directory not empty: .../deployment/web", mxbuild exit 1 or 3, no errors file): +# 17 of 27 failed gates, against 0 of 22 gates run in watcher-free copies of the same project. +# +# Chosen fix: pause the watcher for the length of the gate (SIGSTOP, then SIGCONT). mxbuild +# has no option that moves the deploy target's output out of /deployment, so a +# separate output dir is not available; a pause needs no mxbuild support and no change to the +# watcher. Only the watcher process itself is stopped. Its runtime keeps serving, and a rebuild +# the watcher already started before the pause can still collide, which is what the retry below +# is for. exec.sh resumes the watcher in its EXIT trap too, so an interrupted run does not leave +# it stopped; after a SIGKILL of exec.sh itself, run `kill -CONT ` by hand. +# --------------------------------------------------------------------------- +MXTK_PAUSED_PIDS="" +MXTK_GATE_RETRY_NOTE="" + +# mxtk_watcher_pids - pids of `mxcli run --local --watch` that belong to this +# project: the command line names the project root, or the process runs with it as its cwd. +mxtk_watcher_pids() { + local _root _pid _cmd _cwd + _root="$(cd "$1" 2>/dev/null && pwd -P)" || return 0 + for _pid in $(pgrep -f 'run --local' 2>/dev/null || true); do # portability-ok: pgrep -f ships with Linux procps and macOS; Windows has no watcher to pause here + [ "$_pid" = "$$" ] && continue + _cmd="$(ps -o command= -p "$_pid" 2>/dev/null)" || continue + case "$_cmd" in *mxcli*"run --local"*--watch*) ;; *) continue ;; esac + _cwd="$(readlink "/proc/$_pid/cwd" 2>/dev/null || lsof -a -p "$_pid" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')" + case "$_cmd" in *"$_root"*) echo "$_pid"; continue ;; esac + [ -n "$_cwd" ] && [ "$_cwd" = "$_root" ] && echo "$_pid" + done + return 0 +} + +# mxtk_pause_watchers - SIGSTOP every watcher of this project, remember the pids. +mxtk_pause_watchers() { + local _pid + MXTK_PAUSED_PIDS="" + for _pid in $(mxtk_watcher_pids "$1"); do + if kill -STOP "$_pid" 2>/dev/null; then + MXTK_PAUSED_PIDS="$MXTK_PAUSED_PIDS $_pid" + echo " → paused the run --local --watch watcher (pid $_pid) for the gate; it resumes right after" >&2 + fi + done + return 0 +} + +# mxtk_resume_watchers - SIGCONT whatever mxtk_pause_watchers stopped. Safe to call twice. +mxtk_resume_watchers() { + local _pid + for _pid in $MXTK_PAUSED_PIDS; do kill -CONT "$_pid" 2>/dev/null || true; done + MXTK_PAUSED_PIDS="" + return 0 +} + +# mxtk_gate_mxbuild - the exec gate's way to run +# mxbuild: mxtk_mxbuild_error_count with this project's watcher paused around it, and ONE +# retry when mxbuild exits 1 or 3 and left no sign of a model problem (no CE code in its +# output, nothing in the errors file). That shape is an infrastructure failure, not a verdict +# on the model; a real model error always names a CE code. An errors file with content (for +# example a version mismatch in errors[]) is deterministic and is not retried. +# Prints nothing on stdout (callers discard the count); sets the same MXTK_MXBUILD_* globals as +# mxtk_mxbuild_error_count, plus MXTK_GATE_RETRY_NOTE ("retry after infra exit" or empty). +# Returns mxtk_mxbuild_error_count's code for the final attempt. +mxtk_gate_mxbuild() { + local _mpr="$1" _timeout="${2:-0}" _ef="$3" _rc=0 + MXTK_GATE_RETRY_NOTE="" + mxtk_pause_watchers "${PROJECT_ROOT:-$(dirname "$_mpr")}" + mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$? + case "${MXTK_MXBUILD_EXIT:-}" in + 1|3) + if [ ! -s "$_ef" ] && ! printf '%s\n' "${MXTK_MXBUILD_OUT:-}" | grep -Eq 'CE[0-9]{4}'; then + echo " ⚠ retry after infra exit: mxbuild exited $MXTK_MXBUILD_EXIT with no CE code and no errors file; building once more" >&2 + MXTK_GATE_RETRY_NOTE="retry after infra exit" + : > "$_ef" + _rc=0 + mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$? + fi ;; + esac + mxtk_resume_watchers + return "$_rc" +} + # --------------------------------------------------------------------------- # mxtk_mxbuild_error_count — run mxbuild once, print its Error-severity count. # diff --git a/project-bin/exec.sh b/project-bin/exec.sh index 352f44ea..ef0a1c62 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -437,6 +437,7 @@ BUILD_LOG="$PROJECT_ROOT/docs/BUILD-LOG.md" GATE_STATE="not-run" # not-run | skipped | unverified | pass | fail GATE_ERRORS="?" # mxbuild error count behind a pass (0 = clean); recorded in the stamp MXB_WHY="" # mxbuild's errors[] reason when it exited without checking the model +GATE_NOTE="" # "retry after infra exit" when a gate mxbuild was run twice # ── Exec approval (auto records itself) ────────────────────────────────────── # bin/exec-approval.sh decides ask vs auto; the ASKING happens before exec.sh ever runs (in @@ -490,7 +491,7 @@ HDR2 # The gate cell is read from GATE_STATE rather than passed in, so no caller # can omit it — a blank cell in this table reads as "fine", which is the exact # false-green the gate exists to prevent. - _detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}" + _detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${GATE_NOTE:+ · $GATE_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}" printf '| %s | `%s` | %s | %s | %s |\n' \ "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$LOG_NAME" \ "${GATE_STATE:-not-run}" "$1" "$_detail" >> "$BUILD_LOG" @@ -708,7 +709,7 @@ if [ "$LOCK_TAKEN" != 1 ]; then echo " → Wait for the other exec to finish, then re-run. Nothing was written." exit 1 fi -trap 'exec_lock_release' EXIT +trap 'mxtk_resume_watchers; exec_lock_release' EXIT # 3. A raw `mxcli exec` elsewhere on the machine is NOT this project's business. The old guard # here (`pgrep -fl "mxcli exec"`) matched every exec on the machine, so a test run on a @@ -1353,7 +1354,8 @@ if [ "${SKIP_BASELINE:-0}" != "1" ] && [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]; # can drift apart. Its own printed count is discarded here: this call site # already has err_set/err_count/err_codes on hand and needs BASELINE_SET # (the message set, not just a count) for the delta-gate comparison below. - mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true + mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true + [ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE" # Studio Pro 11's mxbuild (its --help says so) writes --write-errors "only if the # project has errors": on a clean model the mktemp'd file stays EMPTY, json.load # fails, and this read "Model ALREADY has ? error(s) [?]" on every clean run @@ -1633,7 +1635,8 @@ if [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]; then # discarded here: this call site keeps its own file-presence/exit-code # branching below unchanged, computing CE_COUNT via err_count once it knows # which branch applies. - mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true + mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true + [ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE" MXBUILD_EXIT="$MXTK_MXBUILD_EXIT" MXBUILD_OUT="$MXTK_MXBUILD_OUT" MXB_WHY="${MXTK_MXBUILD_WHY:-}" @@ -1751,7 +1754,7 @@ EOF BASE_ERRS=$(mktemp /tmp/mxbuild-base.XXXXXX) # Same shared run+capture as the pre-flight baseline and the main gate # above (mxtk_mxbuild_error_count, _common.sh) — one implementation. - mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true + mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true # Same empty-file rule as the pre-flight baseline: SP 11 mxbuild writes no # errors file on a clean build, and that plus exit 0 is a verified 0. if [ ! -s "$BASE_ERRS" ] && [ "$MXTK_MXBUILD_EXIT" -eq 0 ]; then diff --git a/tests/wave2/test-exec-gate-retry.sh b/tests/wave2/test-exec-gate-retry.sh new file mode 100644 index 00000000..a9cdac6a --- /dev/null +++ b/tests/wave2/test-exec-gate-retry.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# test-exec-gate-retry.sh - exec.sh's gate when mxbuild collides with a live watcher, or fails +# for a reason that is not the model. +# +# usage: bash tests/wave2/test-exec-gate-retry.sh [path-to-exec.sh] +# +# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live +# `mxcli run --local --watch` on the same project both write deployment/. 17 of 27 failed gates +# were "gate could not run" (mxbuild exit 1, no CE code, no errors file); 0 of 22 in watcher-free +# copies. Now: the project's watcher is paused (SIGSTOP/SIGCONT) while the gate runs, and an exit +# 1 or 3 with no CE code and no errors file is retried once. +# +# A infra exit once, then clean: retried, write kept, "retry after infra exit" in output + row +# B infra exit twice: the #424 rollback still applies to the final failure (model identical) +# C a real CE error list (exit 1): NOT retried, normal gate-failed rollback [guard] +# D a live watcher of this project is stopped while mxbuild runs and running again after +# +# Synthetic only: throwaway git repo, stub mxcli, stub mxbuild, stub java. Nothing real runs. +set -uo pipefail + +TOOLKIT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SUBJECT="${1:-$TOOLKIT/project-bin/exec.sh}" +case "$SUBJECT" in /*) ;; *) SUBJECT="$PWD/$SUBJECT" ;; esac +WORK="$(cd "$(mktemp -d "${TMPDIR:-/tmp}/exec-retry.XXXXXX")" && pwd -P)" || exit 2 +WPID="" +trap '[ -n "$WPID" ] && { pkill -KILL -P "$WPID" 2>/dev/null; kill -KILL "$WPID" 2>/dev/null; }; rm -rf "$WORK"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok - $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL - $1${2:+: $2}"; } +echo "== subject: $SUBJECT" + +P="$WORK/proj" +mkdir -p "$P/bin" "$P/mprcontents" "$P/mdlsource" "$WORK/jdk/bin" +printf 'not a real model\n' > "$P/Fixture.mpr" +printf 'bson\n' > "$P/mprcontents/a.mxunit" +printf 'CREATE MODULE "Nope";\n' > "$P/mdlsource/test.mdl" +cp "$TOOLKIT/project-bin/_common.sh" "$TOOLKIT/project-bin/restore-mpr.sh" "$P/bin/" +cp "$SUBJECT" "$P/bin/exec.sh" +cat > "$P/bin/snapshot-mpr.sh" <<'SNAP' +#!/usr/bin/env bash +set -e +cd "$(dirname "$0")/.." +D=".mpr-snapshots/$(date +%Y%m%d-%H%M%S)-$$-$RANDOM" +mkdir -p "$D"; cp Fixture.mpr "$D/"; cp -r mprcontents "$D/" +echo "Snapshot saved: $D" +SNAP +# The write: `mxcli exec` appends to the model, as a real script would. +cat > "$P/mxcli" <<'MXCLI' +#!/usr/bin/env bash +D="$(cd "$(dirname "$0")" && pwd)" +case "${1:-}" in + check) exit 0 ;; + run) sleep 120; exit 0 ;; + exec) echo "written by the script" >> "$D/Fixture.mpr"; echo "x" >> "$D/mprcontents/a.mxunit"; echo "fake mxcli exec"; exit 0 ;; +esac +exit 0 +MXCLI +chmod +x "$P/bin/"*.sh "$P/mxcli" +# mxbuild: MXB_MODES per call number, comma list: ok | infra (exit 1, no CE code, no errors file) | err (a CE0001 error list) +# MXB_WATCH_PID: record that process's state (ps stat) at the start of every call +cat > "$WORK/mxbuild" <<'MXB' +#!/usr/bin/env bash +n=$(( $(cat "$MXB_N" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$MXB_N" +mode="$(printf '%s\n' "$MXB_MODES" | tr ',' '\n' | sed -n "${n}p")"; mode="${mode:-ok}" +[ -n "${MXB_WATCH_PID:-}" ] && ps -o stat= -p "$MXB_WATCH_PID" >> "$MXB_STATES" +err=""; for a in "$@"; do case "$a" in --write-errors=*) err="${a#--write-errors=}" ;; esac; done +case "$mode" in + ok) [ -n "$err" ] && printf '{"problems":[]}' > "$err"; exit 0 ;; + infra) echo "Directory not empty : deployment/web" >&2; exit 1 ;; + err) [ -n "$err" ] && printf '{"problems":[{"severity":"Error","code":"CE0001","message":"boom","element":{"document":"M.X"}}]}' > "$err"; exit 1 ;; +esac +MXB +printf '#!/usr/bin/env bash\nexit 0\n' > "$WORK/jdk/bin/java" +chmod +x "$WORK/mxbuild" "$WORK/jdk/bin/java" +( cd "$P" && git init -q . && printf '.mpr-snapshots/\ndocs/\n.claude/\n' > .gitignore && git add -A \ + && git -c user.email=t@t -c user.name=t commit -qm fixture ) >/dev/null 2>&1 + +LOG="$P/docs/BUILD-LOG.md" +sum() { cat "$P/Fixture.mpr" "$P/mprcontents/a.mxunit" | cksum; } +BEFORE="$(sum)" # the committed model; run() resets to it +# run