Skip to content

fix(exec): pause the project watcher during the gate and retry once on an infra exit #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MendixMau wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix(exec): pause the project watcher during the gate and retry once on an infra exit #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m

## Unreleased

- fix(exec): **the gate pauses this project's `mxcli run --local --watch` while mxbuild runs and retries once on an infra exit.** The gate's mxbuild and a live watcher both write `deployment/`; in a field migration project (12 modules) 17 of 27 failed gates were "gate could not run" (mxbuild exit 1, no CE code), against 0 of 22 in watcher-free copies. The watcher is stopped (SIGSTOP) for the gate and resumed (SIGCONT, also in the EXIT trap); an exit 1 or 3 with no CE code and no errors file is retried once and the BUILD-LOG row says `retry after infra exit`; a final failure still takes the #424 rollback. Test: `tests/wave2/test-exec-gate-retry.sh` (Refs #441, #442, #171, #285, #424) — field migration project
- learn(unit-test): the unit-test skill names the per-flow `Unit flows:` list next to `Unit modules:`, so whether a flow gets a test reads the same in both places (follow-up to #435) — MendixMau
- learn(load-test): **new on-demand skill for load testing a page without correlating Mendix's client protocol by hand: Playwright records the real `/xas/` calls, a probe maps each opaque `runtimeOperation` to its microflow through Jaeger, and the chosen call is replayed with the session's own cookie and CSRF token.** Ten field traps (HTTP 200 on a failed action, caught REST errors under a green microflow span, the wrong call hammered, a recording that still held the previous page, a tracing check that failed open, a harness that throttled itself, harness vs app limits, requests vs users, sampled trace counts, empty data) and a four-line report that says what the run does not prove. Routed ondemand for lead, test and review; no instrument ships yet and the roadmap's "no load testing" line is reworded. Field evidence: a warehouse demo app, Mendix 11, 16 runs 2026-09-09 to 2026-10-08, bursts of 1000 and 2000 concurrent calls on one session — MendixMau
- new(process-map): **Process map: one swimlane map per business process, each step on its wireframe once screens exist, every open question and ASSUMED decision pinned to the step it is about; drawn twice, a draft at Stage 2 close without screens (pass 1, shown at CAC-3) and the full map at Stage 3 with screens (pass 2, walked at the Stage 3 gate close-out).** `skills/process-map.md` (when, model, anchoring, build, verify by looking) and `project-bin/process-map.js` (`validate`, `anchor`, `build`; Node stdlib, deterministic) with its template. Named "map" because "process" in the progress report means the toolkit's own stages, while this shows the app's business process. With no wireframes, a step may name a screen that is not drawn yet or have none, and a person step stays a person step; once a wireframe exists `validate` fails a step page with no wireframe, a use case on no process and not standalone, a decision with one outcome, two steps in one cell. `anchor` places items mechanically (named microflow, listed or named use case, named screen, overlapping source lines) and keeps judged placements by fingerprint, so a reworded question is unplaced again instead of silently wrong; global buckets are allowed and counted. Wired into the Stage 3 Agent produces and Surface rows, a `process-map` opt-in manifest row (an existing model counts as opting in), the `swimlanes` opt-in row (the model discharges it), CAC-3 and CAC-4, and a check_stage_3 FAIL when the map is older than its model, its anchors or a wireframe. `tests/wave2/test-process-map.sh` on a synthetic order approval fixture with its own committed BRD, covering both passes. Field evidence: a requirements-driven rebuild POC, 2026-10-16, where the reviewer could not tell where the sixty questions fit; the map placed 102 items with 0 unplaced. — MendixMau
Expand Down
84 changes: 84 additions & 0 deletions project-bin/_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,90 @@ find_java_exe() {
esac
}

# ---------------------------------------------------------------------------
# Watcher pause and infra retry around the exec gate's mxbuild.
#
# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live
# `mxcli run --local --watch` on the SAME project both write deployment/. One of the two dies
# part-way ("Directory not empty: .../deployment/web", mxbuild exit 1 or 3, no errors file):
# 17 of 27 failed gates, against 0 of 22 gates run in watcher-free copies of the same project.
#
# Chosen fix: pause the watcher for the length of the gate (SIGSTOP, then SIGCONT). mxbuild
# has no option that moves the deploy target's output out of <project>/deployment, so a
# separate output dir is not available; a pause needs no mxbuild support and no change to the
# watcher. Only the watcher process itself is stopped. Its runtime keeps serving, and a rebuild
# the watcher already started before the pause can still collide, which is what the retry below
# is for. exec.sh resumes the watcher in its EXIT trap too, so an interrupted run does not leave
# it stopped; after a SIGKILL of exec.sh itself, run `kill -CONT <pid>` by hand.
# ---------------------------------------------------------------------------
MXTK_PAUSED_PIDS=""
MXTK_GATE_RETRY_NOTE=""

# mxtk_watcher_pids <project-root> - pids of `mxcli run --local --watch` that belong to this
# project: the command line names the project root, or the process runs with it as its cwd.
mxtk_watcher_pids() {
local _root _pid _cmd _cwd
_root="$(cd "$1" 2>/dev/null && pwd -P)" || return 0
for _pid in $(pgrep -f 'run --local' 2>/dev/null || true); do # portability-ok: pgrep -f ships with Linux procps and macOS; Windows has no watcher to pause here
[ "$_pid" = "$$" ] && continue
_cmd="$(ps -o command= -p "$_pid" 2>/dev/null)" || continue
case "$_cmd" in *mxcli*"run --local"*--watch*) ;; *) continue ;; esac
_cwd="$(readlink "/proc/$_pid/cwd" 2>/dev/null || lsof -a -p "$_pid" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')"
case "$_cmd" in *"$_root"*) echo "$_pid"; continue ;; esac
[ -n "$_cwd" ] && [ "$_cwd" = "$_root" ] && echo "$_pid"
done
return 0
}

# mxtk_pause_watchers <project-root> - SIGSTOP every watcher of this project, remember the pids.
mxtk_pause_watchers() {
local _pid
MXTK_PAUSED_PIDS=""
for _pid in $(mxtk_watcher_pids "$1"); do
if kill -STOP "$_pid" 2>/dev/null; then
MXTK_PAUSED_PIDS="$MXTK_PAUSED_PIDS $_pid"
echo " → paused the run --local --watch watcher (pid $_pid) for the gate; it resumes right after" >&2
fi
done
return 0
}

# mxtk_resume_watchers - SIGCONT whatever mxtk_pause_watchers stopped. Safe to call twice.
mxtk_resume_watchers() {
local _pid
for _pid in $MXTK_PAUSED_PIDS; do kill -CONT "$_pid" 2>/dev/null || true; done
MXTK_PAUSED_PIDS=""
return 0
}

# mxtk_gate_mxbuild <mpr> <timeout-seconds> <errors-file> - the exec gate's way to run
# mxbuild: mxtk_mxbuild_error_count with this project's watcher paused around it, and ONE
# retry when mxbuild exits 1 or 3 and left no sign of a model problem (no CE code in its
# output, nothing in the errors file). That shape is an infrastructure failure, not a verdict
# on the model; a real model error always names a CE code. An errors file with content (for
# example a version mismatch in errors[]) is deterministic and is not retried.
# Prints nothing on stdout (callers discard the count); sets the same MXTK_MXBUILD_* globals as
# mxtk_mxbuild_error_count, plus MXTK_GATE_RETRY_NOTE ("retry after infra exit" or empty).
# Returns mxtk_mxbuild_error_count's code for the final attempt.
mxtk_gate_mxbuild() {
local _mpr="$1" _timeout="${2:-0}" _ef="$3" _rc=0
MXTK_GATE_RETRY_NOTE=""
mxtk_pause_watchers "${PROJECT_ROOT:-$(dirname "$_mpr")}"
mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
case "${MXTK_MXBUILD_EXIT:-}" in
1|3)
if [ ! -s "$_ef" ] && ! printf '%s\n' "${MXTK_MXBUILD_OUT:-}" | grep -Eq 'CE[0-9]{4}'; then
echo " ⚠ retry after infra exit: mxbuild exited $MXTK_MXBUILD_EXIT with no CE code and no errors file; building once more" >&2
MXTK_GATE_RETRY_NOTE="retry after infra exit"
: > "$_ef"
_rc=0
mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
fi ;;
esac
mxtk_resume_watchers
return "$_rc"
}

# ---------------------------------------------------------------------------
# mxtk_mxbuild_error_count — run mxbuild once, print its Error-severity count.
#
Expand Down
13 changes: 8 additions & 5 deletions project-bin/exec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,7 @@ BUILD_LOG="$PROJECT_ROOT/docs/BUILD-LOG.md"
GATE_STATE="not-run" # not-run | skipped | unverified | pass | fail
GATE_ERRORS="?" # mxbuild error count behind a pass (0 = clean); recorded in the stamp
MXB_WHY="" # mxbuild's errors[] reason when it exited without checking the model
GATE_NOTE="" # "retry after infra exit" when a gate mxbuild was run twice

# ── Exec approval (auto records itself) ──────────────────────────────────────
# bin/exec-approval.sh decides ask vs auto; the ASKING happens before exec.sh ever runs (in
Expand Down Expand Up @@ -490,7 +491,7 @@ HDR2
# The gate cell is read from GATE_STATE rather than passed in, so no caller
# can omit it — a blank cell in this table reads as "fine", which is the exact
# false-green the gate exists to prevent.
_detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
_detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${GATE_NOTE:+ · $GATE_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
printf '| %s | `%s` | %s | %s | %s |\n' \
"$(date '+%Y-%m-%dT%H:%M:%S%z')" "$LOG_NAME" \
"${GATE_STATE:-not-run}" "$1" "$_detail" >> "$BUILD_LOG"
Expand Down Expand Up @@ -708,7 +709,7 @@ if [ "$LOCK_TAKEN" != 1 ]; then
echo " → Wait for the other exec to finish, then re-run. Nothing was written."
exit 1
fi
trap 'exec_lock_release' EXIT
trap 'mxtk_resume_watchers; exec_lock_release' EXIT

# 3. A raw `mxcli exec` elsewhere on the machine is NOT this project's business. The old guard
# here (`pgrep -fl "mxcli exec"`) matched every exec on the machine, so a test run on a
Expand Down Expand Up @@ -1353,7 +1354,8 @@ if [ "${SKIP_BASELINE:-0}" != "1" ] && [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ];
# can drift apart. Its own printed count is discarded here: this call site
# already has err_set/err_count/err_codes on hand and needs BASELINE_SET
# (the message set, not just a count) for the delta-gate comparison below.
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
[ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
# Studio Pro 11's mxbuild (its --help says so) writes --write-errors "only if the
# project has errors": on a clean model the mktemp'd file stays EMPTY, json.load
# fails, and this read "Model ALREADY has ? error(s) [?]" on every clean run
Expand Down Expand Up @@ -1633,7 +1635,8 @@ if [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]; then
# discarded here: this call site keeps its own file-presence/exit-code
# branching below unchanged, computing CE_COUNT via err_count once it knows
# which branch applies.
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
[ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
MXBUILD_EXIT="$MXTK_MXBUILD_EXIT"
MXBUILD_OUT="$MXTK_MXBUILD_OUT"
MXB_WHY="${MXTK_MXBUILD_WHY:-}"
Expand Down Expand Up @@ -1751,7 +1754,7 @@ EOF
BASE_ERRS=$(mktemp /tmp/mxbuild-base.XXXXXX)
# Same shared run+capture as the pre-flight baseline and the main gate
# above (mxtk_mxbuild_error_count, _common.sh) — one implementation.
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
# Same empty-file rule as the pre-flight baseline: SP 11 mxbuild writes no
# errors file on a clean build, and that plus exit 0 is a verified 0.
if [ ! -s "$BASE_ERRS" ] && [ "$MXTK_MXBUILD_EXIT" -eq 0 ]; then
Expand Down
127 changes: 127 additions & 0 deletions tests/wave2/test-exec-gate-retry.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
#!/usr/bin/env bash
# test-exec-gate-retry.sh - exec.sh's gate when mxbuild collides with a live watcher, or fails
# for a reason that is not the model.
#
# usage: bash tests/wave2/test-exec-gate-retry.sh [path-to-exec.sh]
#
# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live
# `mxcli run --local --watch` on the same project both write deployment/. 17 of 27 failed gates
# were "gate could not run" (mxbuild exit 1, no CE code, no errors file); 0 of 22 in watcher-free
# copies. Now: the project's watcher is paused (SIGSTOP/SIGCONT) while the gate runs, and an exit
# 1 or 3 with no CE code and no errors file is retried once.
#
# A infra exit once, then clean: retried, write kept, "retry after infra exit" in output + row
# B infra exit twice: the #424 rollback still applies to the final failure (model identical)
# C a real CE error list (exit 1): NOT retried, normal gate-failed rollback [guard]
# D a live watcher of this project is stopped while mxbuild runs and running again after
#
# Synthetic only: throwaway git repo, stub mxcli, stub mxbuild, stub java. Nothing real runs.
set -uo pipefail

TOOLKIT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
SUBJECT="${1:-$TOOLKIT/project-bin/exec.sh}"
case "$SUBJECT" in /*) ;; *) SUBJECT="$PWD/$SUBJECT" ;; esac
WORK="$(cd "$(mktemp -d "${TMPDIR:-/tmp}/exec-retry.XXXXXX")" && pwd -P)" || exit 2
WPID=""
trap '[ -n "$WPID" ] && { pkill -KILL -P "$WPID" 2>/dev/null; kill -KILL "$WPID" 2>/dev/null; }; rm -rf "$WORK"' EXIT
PASS=0; FAIL=0
ok() { PASS=$((PASS+1)); echo " ok - $1"; }
bad() { FAIL=$((FAIL+1)); echo " FAIL - $1${2:+: $2}"; }
echo "== subject: $SUBJECT"

P="$WORK/proj"
mkdir -p "$P/bin" "$P/mprcontents" "$P/mdlsource" "$WORK/jdk/bin"
printf 'not a real model\n' > "$P/Fixture.mpr"
printf 'bson\n' > "$P/mprcontents/a.mxunit"
printf 'CREATE MODULE "Nope";\n' > "$P/mdlsource/test.mdl"
cp "$TOOLKIT/project-bin/_common.sh" "$TOOLKIT/project-bin/restore-mpr.sh" "$P/bin/"
cp "$SUBJECT" "$P/bin/exec.sh"
cat > "$P/bin/snapshot-mpr.sh" <<'SNAP'
#!/usr/bin/env bash
set -e
cd "$(dirname "$0")/.."
D=".mpr-snapshots/$(date +%Y%m%d-%H%M%S)-$$-$RANDOM"
mkdir -p "$D"; cp Fixture.mpr "$D/"; cp -r mprcontents "$D/"
echo "Snapshot saved: $D"
SNAP
# The write: `mxcli exec` appends to the model, as a real script would.
cat > "$P/mxcli" <<'MXCLI'
#!/usr/bin/env bash
D="$(cd "$(dirname "$0")" && pwd)"
case "${1:-}" in
check) exit 0 ;;
run) sleep 120; exit 0 ;;
exec) echo "written by the script" >> "$D/Fixture.mpr"; echo "x" >> "$D/mprcontents/a.mxunit"; echo "fake mxcli exec"; exit 0 ;;
esac
exit 0
MXCLI
chmod +x "$P/bin/"*.sh "$P/mxcli"
# mxbuild: MXB_MODES per call number, comma list: ok | infra (exit 1, no CE code, no errors file) | err (a CE0001 error list)
# MXB_WATCH_PID: record that process's state (ps stat) at the start of every call
cat > "$WORK/mxbuild" <<'MXB'
#!/usr/bin/env bash
n=$(( $(cat "$MXB_N" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$MXB_N"
mode="$(printf '%s\n' "$MXB_MODES" | tr ',' '\n' | sed -n "${n}p")"; mode="${mode:-ok}"
[ -n "${MXB_WATCH_PID:-}" ] && ps -o stat= -p "$MXB_WATCH_PID" >> "$MXB_STATES"
err=""; for a in "$@"; do case "$a" in --write-errors=*) err="${a#--write-errors=}" ;; esac; done
case "$mode" in
ok) [ -n "$err" ] && printf '{"problems":[]}' > "$err"; exit 0 ;;
infra) echo "Directory not empty : deployment/web" >&2; exit 1 ;;
err) [ -n "$err" ] && printf '{"problems":[{"severity":"Error","code":"CE0001","message":"boom","element":{"document":"M.X"}}]}' > "$err"; exit 1 ;;
esac
MXB
printf '#!/usr/bin/env bash\nexit 0\n' > "$WORK/jdk/bin/java"
chmod +x "$WORK/mxbuild" "$WORK/jdk/bin/java"
( cd "$P" && git init -q . && printf '.mpr-snapshots/\ndocs/\n.claude/\n' > .gitignore && git add -A \
&& git -c user.email=t@t -c user.name=t commit -qm fixture ) >/dev/null 2>&1

LOG="$P/docs/BUILD-LOG.md"
sum() { cat "$P/Fixture.mpr" "$P/mprcontents/a.mxunit" | cksum; }
BEFORE="$(sum)" # the committed model; run() resets to it
# run <label> <modes> - resets the model, runs the default path, prints the exit code
run() {
( cd "$P" && git checkout -q -- Fixture.mpr mprcontents && rm -rf .mpr-snapshots docs .claude ) >/dev/null 2>&1
rm -f "$WORK/n"; mkdir -p "$WORK/tmp"
( cd "$P" && TMPDIR="$WORK/tmp" SP_RESTART=0 MXTK_NO_INSTALL=1 MXB_N="$WORK/n" MXB_MODES="$2" \
MXBUILD_PATH="$WORK/mxbuild" JAVA_HOME="$WORK/jdk" ./bin/exec.sh mdlsource/test.mdl ) >"$WORK/out.$1" 2>&1
echo $?
}
lastrow() { tail -1 "$LOG" 2>/dev/null; }
kept() { grep -q 'written by the script' "$P/Fixture.mpr"; }

echo "== A: infra exit once, then clean: retried, kept =="
RC=$(run A ok,infra,ok)
[ "$RC" -eq 0 ] && kept && ok "exit 0, write kept" || bad "A rc=$RC" "$(tail -8 "$WORK/out.A")"
[ "$(cat "$WORK/n")" -eq 3 ] && ok "mxbuild ran 3 times (baseline, gate, retry)" || bad "A: mxbuild calls" "$(cat "$WORK/n")"
grep -q 'retry after infra exit' "$WORK/out.A" && ok "says retry after infra exit" || bad "A: no retry message" "$(tail -10 "$WORK/out.A")"
lastrow | grep -q 'retry after infra exit' && lastrow | grep -q 'applied' && ok "applied row carries the retry note" || bad "A: row" "$(lastrow)"

echo "== B: infra exit twice: the #424 rollback applies to the final failure =="
RC=$(run B ok,infra,infra)
[ "$(cat "$WORK/n")" -eq 3 ] && ok "retried exactly once" || bad "B: mxbuild calls" "$(cat "$WORK/n")"
kept && bad "B: the write was KEPT after the final failure" || ok "write rolled back"
[ "$(sum)" = "$BEFORE" ] && ok "model identical to before the exec" || bad "B: model differs from before"
lastrow | grep -q 'gate could not run.*script rolled back' && lastrow | grep -q 'retry after infra exit' \
&& ok "row says rolled back, with the retry note" || bad "B: row" "$(lastrow)"

echo "== C: a real CE error list: no retry [guard] =="
RC=$(run C ok,err)
[ "$(cat "$WORK/n")" -eq 3 ] && ok "no retry: baseline, gate, whose-error rebuild only" || bad "C: mxbuild calls" "$(cat "$WORK/n")"
grep -q 'retry after infra exit' "$WORK/out.C" && bad "C: retried a model error" || ok "no retry message"
lastrow | grep -q 'gate failed' && ok "gate failed row" || bad "C: row" "$(lastrow)"

echo "== D: a live watcher of this project is paused during mxbuild =="
( cd "$P" && exec ./mxcli run --local --watch -p Fixture.mpr ) >/dev/null 2>&1 &
WPID=$!
sleep 1
MXB_STATES="$WORK/states"; : > "$MXB_STATES"
export MXB_WATCH_PID="$WPID" MXB_STATES
RC=$(run D ok,ok)
unset MXB_WATCH_PID
[ "$RC" -eq 0 ] && kept && ok "exit 0, write kept" || bad "D rc=$RC" "$(tail -8 "$WORK/out.D")"
[ "$(wc -l < "$MXB_STATES" | tr -d ' ')" -eq 2 ] && ! grep -qv 'T' "$MXB_STATES" && ok "watcher stopped (T) during both mxbuild runs" || bad "D: states" "$(cat "$MXB_STATES")"
case "$(ps -o stat= -p "$WPID" 2>/dev/null)" in T*) bad "D: watcher still stopped after exec" ;; "") bad "D: watcher is gone" ;; *) ok "watcher running again after exec" ;; esac

echo ""
echo "PASS: $PASS FAIL: $FAIL"
[ "$FAIL" -eq 0 ]
Loading