From ae897b48ab21cdc5d8a66d3f3bf0d3f8b221f763 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:25:26 +0000 Subject: [PATCH 1/3] new(app-facts): incremental loop sweep and app-map freshness in status (#297) The loop sweep re-described every loop-containing microflow on every run: 1,917 describes in 2 h 45 at 6 workers on a 17,500-unit app (about 31 s each), while about 80 units had changed between rounds. - manifest.json stores a {UnitID: ContentsHash} snapshot of the .mpr Unit table and an mdl_index of the unit state each mdl/ file came from. - A refresh describes only new, changed or moved candidates, keeps the rest and deletes outputs of removed units. Full sweep on first run, --full, mxcli version change, or an unreadable Unit table. - UnitID -> name via mx-metadata-note.py units (shared Model base with the History-note code, no duplicate walk). The hash is computed from the .mxunit bytes: one stale Unit-table row was found in a field app's history. - bin/status.sh: "app map: N units changed since " / "fresh". - skills/app-analysis.md: measured cost by app size. Fixture tests/wave2/test-app-facts-incremental.sh (92 checks, both layouts); bench tests/wave2/bench-app-sweep.sh (2,000 units, 1 % changed: 20 describes vs 2,000). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 1 + bin/status.sh | 13 +- project-bin/app-facts.sh | 202 +++++++++++++-- project-bin/mx-metadata-note.py | 158 ++++++++++-- skills/app-analysis.md | 17 +- tests/wave2/bench-app-sweep.sh | 54 ++++ .../fixtures/app-facts-incremental/mkmodel.py | 131 ++++++++++ .../app-facts-incremental/mxcli-stub.sh | 17 ++ tests/wave2/test-app-facts-incremental.sh | 242 ++++++++++++++++++ 9 files changed, 787 insertions(+), 48 deletions(-) create mode 100755 tests/wave2/bench-app-sweep.sh create mode 100755 tests/wave2/fixtures/app-facts-incremental/mkmodel.py create mode 100755 tests/wave2/fixtures/app-facts-incremental/mxcli-stub.sh create mode 100755 tests/wave2/test-app-facts-incremental.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index ff60ea83..578e16da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- new(app-facts): **the app map's loop sweep is incremental, and `bin/status.sh` says how stale the map is** ([#297](https://github.com/MendixMau/mxcli-project-toolkit/issues/297)). Every run re-described every loop-containing microflow: on a 17,500-unit app that was 1,917 describes in 2 h 45 at 6 workers (about 31 s each, not the 1-2 s `app-analysis.md` promised), while about 80 units had changed between rounds. `manifest.json` now stores a `{UnitID: ContentsHash}` snapshot of the `.mpr` Unit table and an `mdl_index` recording the unit state each `mdl/` file was described from. A refresh describes only candidates whose hash, module or folder changed, or that have no output, keeps the rest and deletes outputs of removed or loop-free units. The sweep is full on the first run, with `--full`, after an mxcli version change, or when the Unit table cannot be read (MPR v1); `sections.loops.sweep` says which and why. UnitID-to-name mapping is `mx-metadata-note.py units`, the Studio Pro History note's own walk, refactored into a shared `Model` base for git revisions and the working tree. The hash is taken from the `.mxunit` bytes in the column's format: on a field app's history the column matched the file for all 2,337 units at HEAD, but one row was stale at 1 of 8 committed states, which would have hidden a change. `bin/status.sh` prints `app map: N units changed since ` or `app map: fresh` when a map exists (`app-facts.sh --freshness`, no mxcli needed), and nothing otherwise. `app-analysis.md` gives the measured cost by app size. Fixture `tests/wave2/test-app-facts-incremental.sh`, 92 checks on both layouts with captured template units and a stub mxcli: all describes on a first run, 0 when nothing changed, 1 for a changed, moved, or stale-row unit, a removed unit's output deleted, a failed describe retried, outputs byte-identical to `--full` and to a fresh full run, and the status line; 66 checks fail against master. `tests/wave2/bench-app-sweep.sh`, 2,000 units with 1 % changed: 20 describes incremental against 2,000 full. Field evidence: the unit mapping ran on a 2,337-unit Mendix app (876 microflows named, no duplicates, 1.2 s); no run against a real mxcli describe yet. Not built: replacing describe with catalog queries (mxcli v0.25 `activities` with loop depth), NEEDS-FIELD-DATA, a real v0.25 catalog capture to check parity with the parser. — jopterhorst (#297) - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/bin/status.sh b/bin/status.sh index fd11576d..cd648493 100755 --- a/bin/status.sh +++ b/bin/status.sh @@ -149,6 +149,16 @@ if [ -f "$MS_FILE" ]; then END { printf "manual steps pending %d of %d", p, t }' "$MS_FILE")" fi +# --- app map freshness (#297) -------------------------------------------------------------- +# The map (skills/app-analysis.md) costs hours on a large model, so how stale it is matters: +# app-facts.sh --freshness compares the .mpr Unit table now with the {UnitID: ContentsHash} +# snapshot the last map stored. No map, no line — the screen is unchanged for projects without one. +APPMAP="" +if [ -f "$PROJECT_DIR/analysis/app-facts/manifest.json" ]; then + APPMAP="$(PROJECT_ROOT="$PROJECT_DIR" bash "$TOOLKIT_ROOT/project-bin/app-facts.sh" --freshness \ + --facts-dir "$PROJECT_DIR/analysis/app-facts" 2>/dev/null | head -1)" +fi + # --- doctor receipt ------------------------------------------------------------------------- DR="$PROJECT_DIR/.claude/.doctor-receipt" if [ -f "$DR" ]; then @@ -267,7 +277,7 @@ done if [ "$BRIEF" = 1 ]; then echo "WHERE $NAME · $STAGE_LINE${ENTRY:+ · $ENTRY}${ADOPTED:+ · joined at $ADOPTED}" [ -n "$ROUND_LINE" ] && echo "ROUND $ROUND_LINE" - echo "STATE scripts $N_SCRIPTS written / $N_PASS gate-pass / $N_DONE done- · modules opened $MOD_OPENED${SKELETON:+ · skeleton $SKELETON} · UNSYNCED $UNSYNCED · open questions $OPEN_Q${MANUAL:+ · $MANUAL}" + echo "STATE scripts $N_SCRIPTS written / $N_PASS gate-pass / $N_DONE done- · modules opened $MOD_OPENED${SKELETON:+ · skeleton $SKELETON} · UNSYNCED $UNSYNCED · open questions $OPEN_Q${MANUAL:+ · $MANUAL}${APPMAP:+ · $APPMAP}" [ -n "$WATCH" ] && echo "WATCH $WATCH" echo "NEXT $NEXT" "$TOOLKIT_ROOT/bin/token-burn.sh" "$PROJECT_DIR" --brief 2>/dev/null @@ -279,6 +289,7 @@ printf '%s\n\n' "$TK" printf 'DONE scripts: %s written, %s gate-pass, %s done- · modules opened: %s%s\n' "$N_SCRIPTS" "$N_PASS" "$N_DONE" "$MOD_OPENED" "${SKELETON:+ · skeleton proven $SKELETON}" printf 'OVERDUE %s · %s · %s · UNSYNCED markers: %s · open questions: %s\n' "$DOCTOR" "$LINT" "${COH:-coherence: cadence script not installed}" "$UNSYNCED" "$OPEN_Q" [ -n "$MANUAL" ] && printf ' %s (docs/manual-steps.md)\n' "$MANUAL" +[ -n "$APPMAP" ] && printf ' %s (refresh: bin/app-facts.sh)\n' "$APPMAP" [ -n "$OB_PENDING" ] && printf ' obligations pending: %s\n' "$OB_PENDING" [ -n "$LAST_FAIL" ] && printf ' last gate FAILED: %s\n' "$LAST_FAIL" [ -n "$WATCH" ] && printf 'WATCH %s\n' "$WATCH" diff --git a/project-bin/app-facts.sh b/project-bin/app-facts.sh index 534d3263..167febfc 100755 --- a/project-bin/app-facts.sh +++ b/project-bin/app-facts.sh @@ -19,8 +19,21 @@ # module pairs. Module-level cycles are computed here, each elementary cycle once. # - Loop BODIES are not in the catalog: activities are a flat list per microflow with no # containment column. `mxcli describe microflow` renders proper `loop ... end loop;` -# blocks in 1-2 s each, so step 3 describes only the microflows the catalog says contain a -# loop and parses those. +# blocks, so step 3 describes only the microflows the catalog says contain a loop and +# parses those. A describe is NOT cheap on a big model: ~31 s each on a 17,500-unit app +# (1,917 described in 2 h 45 at 6 workers, field report #297), because each call loads the +# model. So step 3 is incremental (below). +# +# INCREMENTAL SWEEP (#297). manifest.json keeps a {UnitID: ContentsHash} snapshot of the .mpr +# Unit table (the hash taken from the .mxunit bytes, which the column normally equals; see +# mx-metadata-note.py units for the one stale row measured), and an index of which unit state each mdl/ file was described from. A refresh +# re-describes only a candidate whose unit is new, changed (hash), moved (module/folder path) +# or has no output yet; it keeps every other mdl/ file and deletes the outputs of units that are +# gone or no longer loop. UnitID -> name comes from mx-metadata-note.py (beside this script): +# the same mapping that writes the Studio Pro History note. A full sweep runs when asked +# (--full), on the first run, when the mxcli version changed, or when the unit table cannot be +# read (MPR v1, helper missing); the manifest says which and why. Incremental output is +# byte-identical to a full sweep on the same model (tests/wave2/test-app-facts-incremental.sh). # # Read-only for the model. Runs `mxcli graph-report` (rewrites .mxcli/catalog.db, never the # .mpr) and `mxcli describe` (pure read). Never `exec`, never `apply`. @@ -31,11 +44,14 @@ # bin/app-facts.sh --skip-loops # steps 1-2 only (seconds, not minutes) # bin/app-facts.sh --workers N # parallel describe calls for step 3 (default 4) # bin/app-facts.sh --max-loops N # cap the loop sweep (a first look at a huge app) +# bin/app-facts.sh --full # re-describe every candidate, ignoring the index +# bin/app-facts.sh --freshness # one line: units changed since the last map (status.sh) # bin/app-facts.sh --parse-only DIR # re-parse already-described MDL in DIR/mdl (tests) # # Outputs (all under the facts dir): # manifest.json what ran, versions, counts, timings, and a status per section: -# pass | fault | skipped — a section that did not run is never absent +# pass | fault | skipped — a section that did not run is never absent; +# plus units.snapshot {UnitID: ContentsHash} and mdl_index (incremental) # graph-report.json mxcli's own report, kept verbatim for cross-checking # inventory.json project, modules (kind, sizes), scheduled events # dependencies.json module edges, bidirectional pairs, cycles, god nodes, cohesion, dead @@ -70,6 +86,10 @@ SKIP_LOOPS=0 WORKERS=4 MAX_LOOPS=0 PARSE_ONLY="" +FULL=0 +FRESHNESS=0 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +UNITS_PY="$SCRIPT_DIR/mx-metadata-note.py" while [ $# -gt 0 ]; do case "$1" in --facts-dir) FACTS_DIR="${2:-}"; shift 2 ;; @@ -77,7 +97,9 @@ while [ $# -gt 0 ]; do --workers) WORKERS="${2:-4}"; shift 2 ;; --max-loops) MAX_LOOPS="${2:-0}"; shift 2 ;; --parse-only) PARSE_ONLY="${2:-}"; shift 2 ;; - -h|--help) sed -n '2,45p' "$0"; exit 0 ;; + --full) FULL=1; shift ;; + --freshness) FRESHNESS=1; shift ;; + -h|--help) sed -n '2,62p' "$0"; exit 0 ;; *) echo "app-facts: unknown argument '$1'" >&2; exit 2 ;; esac done @@ -207,6 +229,34 @@ MODEL_DIR="$(cd "$(dirname "$MPR")" && pwd)" MPR_BASE="$(basename "$MPR")" DB="$MODEL_DIR/.mxcli/catalog.db" [ -n "$FACTS_DIR" ] || FACTS_DIR="${PROJECT_ROOT:-.}/analysis/app-facts" + +# --------------------------------------------------------------------------------------------- +# --freshness: one line for bin/status.sh. Compares the Unit table now with the snapshot the +# last map stored. No mxcli, no catalog, writes nothing. Silent when there is no map yet. +# --------------------------------------------------------------------------------------------- +if [ "$FRESHNESS" -eq 1 ]; then + [ -f "$FACTS_DIR/manifest.json" ] || exit 0 + NOW_TSV="$("$PY" "$UNITS_PY" units "$MODEL_DIR/$MPR_BASE" --hashes-only 2>/dev/null)" || NOW_TSV="__unreadable__" + NOW_TSV="$NOW_TSV" "$PY" - "$FACTS_DIR/manifest.json" <<'PY' +import json, os, sys +try: + man = json.load(open(sys.argv[1])) +except (OSError, ValueError): + print("app map: manifest.json unreadable — rerun bin/app-facts.sh"); sys.exit(0) +when = (man.get("generated") or "?")[:16].replace("T", " ") +snap = (man.get("units") or {}).get("snapshot") +now = os.environ["NOW_TSV"] +if not snap: + print("app map: built %s with no unit snapshot — rerun bin/app-facts.sh to track freshness" % when) +elif now == "__unreadable__": + print("app map: built %s; unit table unreadable, freshness unknown" % when) +else: + cur = dict(l.split("\t", 1) for l in now.splitlines() if "\t" in l) + n = sum(1 for u, h in cur.items() if snap.get(u) != h) + sum(1 for u in snap if u not in cur) + print("app map: fresh (built %s)" % when if n == 0 else "app map: %d units changed since %s" % (n, when)) +PY + exit 0 +fi mkdir -p "$FACTS_DIR/mdl" || exit 2 FACTS_DIR="$(cd "$FACTS_DIR" && pwd)" @@ -230,7 +280,8 @@ log() { printf '==> %s\n' "$*"; } # --------------------------------------------------------------------------------------------- # Step 1 — full catalog build, via graph-report (one call: the rebuild AND mxcli's own report). -# First run on a cold catalog: ~20-30 s on 1,000-2,700 microflows, measured. Not a hang. +# First run on a cold catalog: ~20-30 s on 1,000-2,700 microflows, 7 min on a 17,500-unit app +# (#297), measured. Not a hang. # --------------------------------------------------------------------------------------------- log "step 1: full catalog build + mxcli graph-report ($MPR_BASE)" T1=$(date +%s) @@ -511,17 +562,94 @@ PY T2=$(( $(date +%s) - T2 )) # --------------------------------------------------------------------------------------------- -# Step 3 — describe every loop-containing microflow, in parallel, then parse the loop bodies. +# Step 2b — unit snapshot: {UnitID: ContentsHash} from the .mpr Unit table, with each unit's +# type, qualified name and folder path. Seconds, not minutes (0.3 s on 2,337 units). A model +# whose table cannot be read gets a full sweep, never a guessed incremental one. +# --------------------------------------------------------------------------------------------- +UNITS_TSV="$FACTS_DIR/.units.tsv"; UNITS_STATUS="fault"; UNITS_REASON="" +rm -f "$UNITS_TSV" +if [ ! -f "$UNITS_PY" ]; then + UNITS_REASON="mx-metadata-note.py not beside app-facts.sh" +elif "$PY" "$UNITS_PY" units "$MODEL_DIR/$MPR_BASE" > "$UNITS_TSV" 2> "$FACTS_DIR/.units.err" && [ -s "$UNITS_TSV" ]; then + UNITS_STATUS="pass"; UNITS_REASON="$(wc -l < "$UNITS_TSV" | tr -d ' ') units read from the Unit table" + [ -s "$FACTS_DIR/.units.err" ] && UNITS_REASON="$UNITS_REASON; $(tail -1 "$FACTS_DIR/.units.err" | cut -c1-160)" +else + UNITS_REASON="unit table unreadable: $(tail -1 "$FACTS_DIR/.units.err" 2>/dev/null | cut -c1-160)" + rm -f "$UNITS_TSV" +fi +rm -f "$FACTS_DIR/.units.err" + +# --------------------------------------------------------------------------------------------- +# Step 3 — describe the loop-containing microflows whose unit changed since their mdl/ file was +# written (all of them on a full sweep), in parallel, then parse every loop body. # --------------------------------------------------------------------------------------------- LOOP_STATUS="skipped"; LOOP_REASON="--skip-loops"; T3=0; N_CAND=0; N_DESC=0; N_FAILED=0; N_RULES=0 +SWEEP_MODE="none"; SWEEP_REASON="--skip-loops" N_CAND=$(wc -l < "$FACTS_DIR/loops-candidates.tsv" | tr -d ' ') +PLAN="$FACTS_DIR/.sweep-plan.json"; TODO="$FACTS_DIR/.describe-todo.txt" +rm -f "$PLAN" "$TODO" if [ "$SKIP_LOOPS" -eq 0 ]; then T3=$(date +%s) + # The plan: which candidates to describe, which outputs to keep, which to delete. It also + # deletes every mdl/ file that is about to be rewritten or no longer belongs, so a describe + # that fails leaves no stale file behind. + FACTS_DIR="$FACTS_DIR" UNITS_TSV="$UNITS_TSV" FULL="$FULL" MX_VERSION="$MX_VERSION" PLAN="$PLAN" TODO="$TODO" \ + UNITS_REASON="$UNITS_REASON" "$PY" - <<'PY' || exit 2 +import json, os +F = os.environ["FACTS_DIR"]; MDL = os.path.join(F, "mdl") +by_qn = {} +if os.path.exists(os.environ["UNITS_TSV"]): + for line in open(os.environ["UNITS_TSV"], encoding="utf-8"): + p = line.rstrip("\n").split("\t") + if len(p) == 5 and p[3]: + by_qn[p[3]] = {"unit": p[0], "hash": p[1], "path": p[4]} +cands = [] +for line in open(os.path.join(F, "loops-candidates.tsv"), encoding="utf-8"): + p = line.rstrip("\n").split("\t") + if len(p) >= 4 and p[3] != "RULE": + cands.append(p[0]) +try: + prev = json.load(open(os.path.join(F, "manifest.json"))) +except (OSError, ValueError): + prev = {} +prev_idx = prev.get("mdl_index") or {} +if os.environ["FULL"] == "1": + mode, why = "full", "--full" +elif not by_qn: + mode, why = "full", "no unit snapshot (%s)" % os.environ["UNITS_REASON"] +elif not prev_idx: + mode, why = "full", "no mdl_index from an earlier run (first run, or facts written before incremental sweeps)" +elif prev.get("mdl_index_mxcli") != os.environ["MX_VERSION"]: + mode, why = "full", "mxcli changed since the last sweep (%s -> %s)" % (prev.get("mdl_index_mxcli"), os.environ["MX_VERSION"]) +else: + mode, why = "incremental", "unit hashes compared with mdl_index" +want = set(cands) +have = {fn[:-4] for fn in os.listdir(MDL) if fn.endswith(".mdl")} +todo, reused, removed = [], [], [] +for qn in sorted(have): + if mode == "full" or qn not in want: + os.unlink(os.path.join(MDL, qn + ".mdl")) + if qn not in want: removed.append(qn) +for qn in cands: + cur = by_qn.get(qn) + if mode == "incremental" and qn in have and cur is not None and prev_idx.get(qn) == cur: + reused.append(qn); continue + if qn in have and mode == "incremental": + os.unlink(os.path.join(MDL, qn + ".mdl")) + todo.append(qn) +with open(os.environ["TODO"], "w", encoding="utf-8") as fh: + for qn in todo: fh.write(qn + "\n") +json.dump({"mode": mode, "reason": why, "describe": len(todo), "reused": len(reused), "removed": len(removed), + "index": {qn: by_qn[qn] for qn in cands if qn in by_qn}}, open(os.environ["PLAN"], "w")) +PY + SWEEP_MODE="$("$PY" -c 'import json,sys; print(json.load(open(sys.argv[1]))["mode"])' "$PLAN")" + SWEEP_REASON="$("$PY" -c 'import json,sys; print(json.load(open(sys.argv[1]))["reason"])' "$PLAN")" + N_TODO=$(wc -l < "$TODO" | tr -d ' ') if [ "${N_CAND:-0}" -eq 0 ]; then LOOP_STATUS="pass"; LOOP_REASON="no LoopedActivity in the catalog; nothing to describe" echo '{"_meta":{"files":0,"parsed":0,"parse_mismatch":[],"unreadable":[]},"microflows":{}}' > "$FACTS_DIR/loops.json" else - log "step 3: describing $N_CAND loop-containing microflows with $WORKERS workers (~1-2 s each)" + log "step 3: $SWEEP_MODE sweep ($SWEEP_REASON): describing $N_TODO of $N_CAND loop-containing microflows with $WORKERS workers" : > "$FACTS_DIR/describe-failures.txt" # -n 1 with the name as $1, not -I{}: BSD xargs refuses -I when the replacement string # appears more than a few times in the command ("command line cannot be assembled"). @@ -529,11 +657,12 @@ if [ "$SKIP_LOOPS" -eq 0 ]; then # and all three live in microflows_data), and `describe microflow` says "not found" for the # other two. 11 of 425 on the first real run, all nanoflows and rules. N_RULES=$(awk -F'\t' '$4=="RULE"' "$FACTS_DIR/loops-candidates.tsv" | wc -l | tr -d ' ') - awk -F'\t' '$4!="RULE"{print $1}' "$FACTS_DIR/loops-candidates.tsv" | \ + if [ "$N_TODO" -gt 0 ]; then MXCLI="$MXCLI" MODEL_DIR="$MODEL_DIR" MPR_BASE="$MPR_BASE" OUT="$FACTS_DIR" \ - xargs -P "$WORKERS" -n 1 sh -c ' - cd "$MODEL_DIR" && "$MXCLI" describe "$1" -p "$MPR_BASE" > "$OUT/mdl/$1.mdl" 2>/dev/null \ - || { echo "$1" >> "$OUT/describe-failures.txt"; rm -f "$OUT/mdl/$1.mdl"; }' _ + xargs -P "$WORKERS" -n 1 sh -c ' + cd "$MODEL_DIR" && "$MXCLI" describe "$1" -p "$MPR_BASE" > "$OUT/mdl/$1.mdl" 2>/dev/null \ + || { echo "$1" >> "$OUT/describe-failures.txt"; rm -f "$OUT/mdl/$1.mdl"; }' _ < "$TODO" + fi N_DESC=$(ls "$FACTS_DIR"/mdl/*.mdl 2>/dev/null | wc -l | tr -d ' ') N_FAILED=$(wc -l < "$FACTS_DIR/describe-failures.txt" | tr -d ' ') if [ "${N_DESC:-0}" -eq 0 ]; then @@ -541,7 +670,7 @@ if [ "$SKIP_LOOPS" -eq 0 ]; then exit 2 fi "$PY" -c "$PARSER" "$FACTS_DIR/mdl" "$FACTS_DIR/loops-expected.json" "$FACTS_DIR/loops-scheduled.json" > "$FACTS_DIR/loops.json" || exit 2 - LOOP_STATUS="pass"; LOOP_REASON="described $N_DESC of $N_CAND; $N_FAILED failed; $N_RULES rules not describable by this mxcli" + LOOP_STATUS="pass"; LOOP_REASON="described $N_DESC of $N_CAND ($N_TODO this run, $SWEEP_MODE); $N_FAILED failed; $N_RULES rules not describable by this mxcli" # A rule with a loop is a known blind spot (no `describe rule` in mxcli v0.21.0), reported as a # count so the dossier can name it under Method. A microflow or nanoflow that failed is partial. [ "$N_FAILED" -gt 0 ] && LOOP_STATUS="partial" @@ -556,9 +685,38 @@ fi FACTS_DIR="$FACTS_DIR" MX_VERSION="$MX_VERSION" MPR_BASE="$MPR_BASE" DB="$DB" \ T1="$T1" T2="$T2" T3="$T3" TALL="$(( $(date +%s) - START_ALL ))" \ LOOP_STATUS="$LOOP_STATUS" LOOP_REASON="$LOOP_REASON" N_CAND="$N_CAND" N_DESC="$N_DESC" N_FAILED="$N_FAILED" N_RULES="$N_RULES" \ -BUILD_MODE="$BUILD_MODE" N_MODULES="$N_MODULES" N_REFS="$N_REFS" N_ACTS="$N_ACTS" WORKERS="$WORKERS" MAX_LOOPS="$MAX_LOOPS" "$PY" - <<'PY' +BUILD_MODE="$BUILD_MODE" N_MODULES="$N_MODULES" N_REFS="$N_REFS" N_ACTS="$N_ACTS" WORKERS="$WORKERS" MAX_LOOPS="$MAX_LOOPS" \ +UNITS_TSV="$UNITS_TSV" UNITS_STATUS="$UNITS_STATUS" UNITS_REASON="$UNITS_REASON" PLAN="$PLAN" FULL="$FULL" \ +SWEEP_MODE="$SWEEP_MODE" SWEEP_REASON="$SWEEP_REASON" "$PY" - <<'PY' import json, os, datetime, hashlib F = os.environ["FACTS_DIR"] +try: + prev = json.load(open(os.path.join(F, "manifest.json"))) +except (OSError, ValueError): + prev = {} +# Unit snapshot: the state of the model this map was built from. bin/status.sh compares it with +# the Unit table now (app-facts.sh --freshness). +snap = {} +if os.path.exists(os.environ["UNITS_TSV"]): + for line in open(os.environ["UNITS_TSV"], encoding="utf-8"): + p = line.rstrip("\n").split("\t") + if len(p) >= 2: snap[p[0]] = p[1] +prev_snap = (prev.get("units") or {}).get("snapshot") or {} +changed = None +if snap and prev_snap: + changed = {"previous_map": prev.get("generated", ""), + "changed": sum(1 for u, h in snap.items() if u in prev_snap and prev_snap[u] != h), + "added": sum(1 for u in snap if u not in prev_snap), + "removed": sum(1 for u in prev_snap if u not in snap)} +# mdl_index: which unit state each mdl/ file was described from. Only files that exist now are +# indexed, so a failed describe is retried next run. --skip-loops leaves mdl/ alone and carries the +# previous index forward unchanged. +plan = json.load(open(os.environ["PLAN"])) if os.path.exists(os.environ["PLAN"]) else None +if plan is not None: + mdl_index = {qn: v for qn, v in plan["index"].items() if os.path.exists(os.path.join(F, "mdl", qn + ".mdl"))} + mdl_index_mxcli = os.environ["MX_VERSION"] +else: + mdl_index, mdl_index_mxcli = prev.get("mdl_index") or {}, prev.get("mdl_index_mxcli", "") def sha(p): h = hashlib.sha256() with open(p, "rb") as fh: @@ -575,24 +733,38 @@ man = { "mxcli": os.environ["MX_VERSION"], "catalog": {"modules": int(os.environ["N_MODULES"]), "refs": int(os.environ["N_REFS"]), "activities": int(os.environ["N_ACTS"]), "build_mode": os.environ.get("BUILD_MODE", "")}, "timings_s": {"catalog_and_graph_report": int(os.environ["T1"]), "inventory_and_dependencies": int(os.environ["T2"]), "loops": int(os.environ["T3"]), "total": int(os.environ["TALL"])}, - "options": {"workers": int(os.environ["WORKERS"]), "max_loops": int(os.environ["MAX_LOOPS"])}, + "options": {"workers": int(os.environ["WORKERS"]), "max_loops": int(os.environ["MAX_LOOPS"]), "full": os.environ["FULL"] == "1"}, + "units": {"status": os.environ["UNITS_STATUS"], "reason": os.environ["UNITS_REASON"], "count": len(snap), + "source": "the .mpr Unit table; ContentsHash format, computed from each .mxunit's bytes", "changed_since_previous_map": changed, + "snapshot": snap}, + "mdl_index": mdl_index, "mdl_index_mxcli": mdl_index_mxcli, "sections": { "inventory": {"status": "pass", "file": "inventory.json"}, "dependencies": {"status": "pass", "file": "dependencies.json"}, "loops": {"status": os.environ["LOOP_STATUS"], "file": "loops.json", "reason": os.environ["LOOP_REASON"], "candidates": int(os.environ["N_CAND"]), "described": int(os.environ["N_DESC"]), "failed": int(os.environ["N_FAILED"]), "rules_not_describable": int(os.environ["N_RULES"]), "parse_mismatch": mismatch, - "catalog_undercount": int(loops["_meta"].get("catalog_undercount", 0)) if loops else 0}, + "catalog_undercount": int(loops["_meta"].get("catalog_undercount", 0)) if loops else 0, + "sweep": {"mode": os.environ["SWEEP_MODE"], "reason": os.environ["SWEEP_REASON"], + "described_this_run": plan["describe"] if plan else 0, + "reused": plan["reused"] if plan else 0, + "removed_outputs": plan["removed"] if plan else 0}}, "security": {"status": "fault", "reason": "not collected by this version of app-facts.sh"}, "lint_baseline":{"status": "fault", "reason": "run project-bin/lint-gate.sh separately; not folded in yet"}, }, } json.dump(man, open(os.path.join(F, "manifest.json"), "w"), indent=1, sort_keys=True) print("==> facts written to %s (total %ss)" % (F, os.environ["TALL"])) +if changed is not None: + print(" units since the previous map (%s): %d changed, %d added, %d removed" + % (changed["previous_map"][:16].replace("T", " "), changed["changed"], changed["added"], changed["removed"])) if mismatch: print(" NOTE: %d microflow(s) where the parser found FEWER loops than the catalog, see loops.json _meta.parse_mismatch" % mismatch) uc = int(loops["_meta"].get("catalog_undercount", 0)) if loops else 0 if uc: print(" note: %d microflow(s) where the parser found MORE loops than the catalog (nested loops; the catalog holds top-level activities only)" % uc) PY +RC=$? +rm -f "$UNITS_TSV" "$PLAN" "$TODO" +[ "$RC" -eq 0 ] || exit 2 exit 0 diff --git a/project-bin/mx-metadata-note.py b/project-bin/mx-metadata-note.py index 66a19576..452cc0f2 100755 --- a/project-bin/mx-metadata-note.py +++ b/project-bin/mx-metadata-note.py @@ -6,6 +6,7 @@ # bin/mx-metadata-note.py write [] [--force] # write that commit's note # bin/mx-metadata-note.py backfill [] [--dry-run] [--force] # bin/mx-metadata-note.py sync # publish: fetch + merge the remote's notes, then push +# bin/mx-metadata-note.py units [--hashes-only] # working-tree unit list (app-facts.sh) # (git push never carries the notes -- run sync after pushing the branch) # (all take --mpr when there is more than one candidate .mpr) # Run it as "$PY" bin/mx-metadata-note.py, PY from _common.sh's resolve_py -- never through the @@ -37,7 +38,9 @@ # notes backfilled on a 357-commit field app, pushed, and Studio Pro 11.15 History shows them (screenshot # confirmed by the user, 2026-10-12). # -# PRODUCER/CONSUMER. Reads only git objects -- no working tree, no mxcli, no mxbuild. Run by the +# PRODUCER/CONSUMER. Reads only git objects -- no working tree, no mxcli, no mxbuild -- except +# `units`, which reads the working-tree model for project-bin/app-facts.sh (its incremental loop +# sweep and its freshness line in bin/status.sh); one UnitID -> name mapping serves both. Run by the # post-commit / post-rewrite hooks bin/install-project-hooks.sh installs (init-project.sh and # sync-project.sh run that installer). Studio Pro is the consumer. Never blocks a commit: every # failure prints one warning and exits 0, because a missing note costs a History entry and a @@ -45,7 +48,7 @@ # # Stdlib only (sqlite3, json, subprocess): runs under any python3 on mac, Linux and Git Bash. -import json, os, re, sqlite3, struct, subprocess, sys, tempfile +import base64, hashlib, json, os, re, sqlite3, struct, subprocess, sys, tempfile, urllib.parse REF = "mx_metadata" STATUS = {"A": "Added", "M": "Modified", "D": "Deleted"} @@ -110,35 +113,42 @@ def find_mpr(arg=None): return os.path.relpath(real[0], os.path.realpath(top)).replace(os.sep, "/") -class Rev: - """The model as it stood at one commit.""" +def unit_table(path): + """{UnitID: (ContainerID, ContentsHash)} from an .mpr's Unit table, opened read-only.""" + uri = "file:" + urllib.parse.quote(os.path.abspath(path).replace(os.sep, "/"), safe="/:") + "?mode=ro" + db = sqlite3.connect(uri, uri=True) + try: + cols = {r[1] for r in db.execute("pragma table_info(Unit)")} + hcol = "ContentsHash" if "ContentsHash" in cols else "NULL" + return {guid(u): ((guid(c) if c else None), h or "") + for u, c, h in db.execute(f"select UnitID, ContainerID, {hcol} from Unit")} + finally: + db.close() - def __init__(self, rev, mpr): - self.rev, self.mpr = rev, mpr - self.contents = (os.path.dirname(mpr) + "/mprcontents").lstrip("/") - self._units, self._parent = {}, None + +class Model: + """The document tree of one model: the .mpr Unit table plus the unit bodies. Rev reads it + as it stood at a commit (git objects only); Tree reads the working tree (app-facts.sh's + incremental sweep and its freshness line). module_of/qualified_name are the one mapping + from a UnitID to the name Studio Pro and mxcli show.""" + + def __init__(self, mpr): + self.mpr = mpr + self._units, self._table = {}, None + + def table(self): + if self._table is None: + self._table = self._read_table() + return self._table def parent_of(self, uid): - if self._parent is None: - fd, path = tempfile.mkstemp(suffix=".mpr") - try: - with os.fdopen(fd, "wb") as f: - f.write(git("show", f"{self.rev}:{self.mpr}", binary=True)) - db = sqlite3.connect(path) - try: - self._parent = {guid(u): (guid(c) if c else None) - for u, c in db.execute("select UnitID, ContainerID from Unit")} - finally: - db.close() - finally: - os.unlink(path) - return self._parent.get(uid) + row = self.table().get(uid) + return row[0] if row else None def unit(self, uid): if uid not in self._units: - path = f"{self.contents}/{uid[:2]}/{uid[2:4]}/{uid}.mxunit" - r = subprocess.run(["git", "show", f"{self.rev}:{path}"], capture_output=True) - self._units[uid] = top_level_strings(r.stdout) if r.returncode == 0 else {} + b = self._unit_bytes(uid) + self._units[uid] = top_level_strings(b) if b else {} return self._units[uid] def module_of(self, uid): @@ -151,6 +161,96 @@ def module_of(self, uid): cur = self.parent_of(cur) return "" + def folder_path(self, uid): + """Names of the containers between the module and the unit, outermost first.""" + names, seen, cur = [], {uid}, self.parent_of(uid) + while cur and cur not in seen: + seen.add(cur) + d = self.unit(cur) + if d.get("$Type") != "Projects$Folder": + break + names.append(d.get("Name", "")) + cur = self.parent_of(cur) + return list(reversed(names)) + + +class Rev(Model): + """The model as it stood at one commit.""" + + def __init__(self, rev, mpr): + super().__init__(mpr) + self.rev = rev + self.contents = (os.path.dirname(mpr) + "/mprcontents").lstrip("/") + + def _read_table(self): + fd, path = tempfile.mkstemp(suffix=".mpr") + try: + with os.fdopen(fd, "wb") as f: + f.write(git("show", f"{self.rev}:{self.mpr}", binary=True)) + return unit_table(path) + finally: + os.unlink(path) + + def _unit_bytes(self, uid): + path = f"{self.contents}/{uid[:2]}/{uid[2:4]}/{uid}.mxunit" + r = subprocess.run(["git", "show", f"{self.rev}:{path}"], capture_output=True) + return r.stdout if r.returncode == 0 else None + + +class Tree(Model): + """The model in the working tree: and /mprcontents. MPR v2 only -- a v1 + model keeps its units inside the .mpr, and the caller falls back to a full sweep.""" + + def __init__(self, mpr): + super().__init__(mpr) + self.contents = os.path.join(os.path.dirname(os.path.abspath(mpr)), "mprcontents") + if not os.path.isdir(self.contents): + raise SystemExit(f"mx-metadata-note units: {self.contents} not found (MPR v1 model?)") + + def _read_table(self): + return unit_table(self.mpr) + + def _unit_bytes(self, uid): + try: + with open(os.path.join(self.contents, uid[:2], uid[2:4], uid + ".mxunit"), "rb") as f: + return f.read() + except OSError: + return None + + +def units(mpr, hashes_only=False): + """One TSV line per unit of the working-tree model: UnitID, content hash and, unless + hashes_only, UnitType, qualified name (Module.Name, "" for a unit outside a module) and the + folder path (Module/Folder/.../Name). + + The hash has the Unit table's ContentsHash format, base64(sha256()), and + is computed from the bytes. Measured on a field app's history: the column equals that value on + all 2,337 units at HEAD, Studio Pro- and mxcli-written alike, but at one of 8 committed states + one unit's column was stale (its file had changed, its row had not). The bytes are what + `mxcli describe` reads, so they win; the column is the fallback for a unit with no file, and + each disagreement is counted on stderr.""" + t = Tree(mpr) + out, stale = sys.stdout, 0 + for uid in sorted(t.table()): + h = t.table()[uid][1] + b = t._unit_bytes(uid) + if b is not None: + fh = base64.b64encode(hashlib.sha256(b).digest()).decode() + stale += bool(h) and h != fh + h = fh + if hashes_only: + out.write(f"{uid}\t{h}\n") + continue + d = t.unit(uid) + mod, name = t.module_of(uid), d.get("Name", "") + qn = f"{mod}.{name}" if mod and name and d.get("$Type") != "Projects$ModuleImpl" else "" + path = "/".join([mod] + t.folder_path(uid) + [name]) if qn else "" + out.write(f"{uid}\t{h}\t{d.get('$Type', '')}\t{qn}\t{path}\n") + if stale: + print(f"{stale} unit(s) whose Unit-table ContentsHash disagrees with the .mxunit bytes; the bytes were used", + file=sys.stderr) + return 0 + def model_changes(commit, mpr): parents = git("rev-list", "--parents", "-n1", commit).split()[1:] @@ -317,12 +417,18 @@ def main(argv): if "--mpr" in args: i = args.index("--mpr"); mpr_arg = args[i + 1]; del args[i:i + 2] cmd = args.pop(0) if args else "help" + if cmd == "units": + hashes = "--hashes-only" in args + args = [a for a in args if a != "--hashes-only"] + if not args: + raise SystemExit("usage: mx-metadata-note.py units [--hashes-only]") + return units(args[0], hashes) if cmd == "sync": if not args: raise SystemExit("usage: mx-metadata-note.py sync ") return sync(args[0]) if cmd not in ("changes", "write", "backfill"): - print("usage: mx-metadata-note.py changes|write [] [--force] | backfill [] [--dry-run] [--force] | sync [--mpr ]") + print("usage: mx-metadata-note.py changes|write [] [--force] | backfill [] [--dry-run] [--force] | sync [--mpr ] | units [--hashes-only]") return 2 mpr = find_mpr(mpr_arg) if cmd == "changes": diff --git a/skills/app-analysis.md b/skills/app-analysis.md index f633826d..f0e6d890 100644 --- a/skills/app-analysis.md +++ b/skills/app-analysis.md @@ -29,10 +29,14 @@ carry them all across. ## Procedure -1. **Collect.** `bin/app-facts.sh` in the project root. First run on a big app takes minutes: - a full catalog rebuild plus 1-2 s per loop-containing microflow (300 of them at 4 workers - is 2-3 minutes). Exit 2 means the facts are not trustworthy; read the message, fix the - cause, rerun. Do not write a dossier from a run that exited 2. +1. **Collect.** `bin/app-facts.sh` in the project root. The first run describes every + loop-containing microflow, and each describe loads the model, so cost grows with app size. + Measured on a 17,500-unit app (127 modules): catalog 7 min, inventory 2 min, and 1,917 + describes in 2 h 45 at 6 workers, about 31 s each. Tell the user before a first run on a + large app. A refresh re-describes only units whose hash, module or folder changed since + their output was written (`manifest.json` → `sections.loops.sweep`); `--full` forces all. + Exit 2 means the facts are not trustworthy; read the message, fix the cause, rerun. Do not + write a dossier from a run that exited 2. 2. **Read `manifest.json` first.** Check `sections.*.status`. Note `parse_mismatch`, `failed` and `rules_not_describable` under loops: each one is a document the loop section says nothing about. `catalog_undercount` is not in that list; it is the catalog's blind spot. @@ -292,8 +296,9 @@ Refresh at every milestone and before every review, not on every commit. The ins rewrites the catalog, which every other mxcli command shares, so run it when nobody is mid-edit. Commit `architecture/app-dossier.md` and `analysis/app-facts/manifest.json`, `inventory.json`, `dependencies.json`, `loops.json`. Do not commit `analysis/app-facts/mdl/` -(hundreds of files that regenerate in minutes) or the HTML render, unless the project has -decided otherwise. +or the HTML render, unless the project has decided otherwise, but keep `mdl/` on disk between +refreshes: a refresh reuses those files, and a clone without them describes every candidate +again. `bin/status.sh` shows how many units changed since the map was built. ## Related diff --git a/tests/wave2/bench-app-sweep.sh b/tests/wave2/bench-app-sweep.sh new file mode 100755 index 00000000..9b617194 --- /dev/null +++ b/tests/wave2/bench-app-sweep.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# bench-app-sweep.sh — describes per refresh of the app-facts loop sweep, full vs incremental (#297). +# +# Usage: bash tests/wave2/bench-app-sweep.sh [path-to-app-facts.sh] [units] [percent-changed] +# defaults: project-bin/app-facts.sh, 2000 units, 1 % changed +# +# Builds a synthetic model of N microflow units, every one loop-containing (the worst case: one +# describe per unit), through fixtures/app-facts-incremental/ (a stub mxcli that counts describes), +# then runs: a first map (full), a refresh with nothing changed, a refresh with P % of the units +# changed, and a --full refresh. The count of describes is the number that matters: on a real +# model each one costs seconds to tens of seconds (~31 s each measured on a 17,500-unit app), and +# the stub's wall time here says nothing about that. Exit 1 when the incremental refresh describes +# anything other than exactly the changed units. + +set -uo pipefail +TK="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +AF="${1:-$TK/project-bin/app-facts.sh}" +case "$AF" in /*) ;; *) AF="$PWD/$AF" ;; esac +N="${2:-2000}"; PCT="${3:-1}" +FX="$TK/tests/wave2/fixtures/app-facts-incremental" +command -v sqlite3 >/dev/null || { echo "SKIP: sqlite3 not on PATH"; exit 0; } +. "$TK/bin/lib/portable.sh"; require_py + +W="$(mktemp -d "${TMPDIR:-/tmp}/af-bench.XXXXXX")" || exit 2 +trap 'rm -rf "$W"' EXIT +P="$W/proj"; mkdir -p "$P" +export PREP_DB="$W/prep.db" DESCRIBE_LOG="$W/describe.log" STUB_VERSION="mxcli stub 1.0" +CHANGED=$(( N * PCT / 100 )) + +spec() { # spec + awk -v n="$N" -v c="$CHANGED" -v v="$1" 'BEGIN { for (i = 1; i <= n; i++) printf "SUB_%05d\t%s\t-\t1\n", i, (i <= c ? v : 1) }' +} +build() { spec "$1" > "$W/spec.tsv"; "$PY" "$FX/mkmodel.py" "$P" "$PREP_DB" "$W/spec.tsv" || exit 2; } +run() { + : > "$DESCRIBE_LOG" + local t0; t0=$(date +%s) + ( cd "$P" && PROJECT_ROOT="$P" MXCLI="$FX/mxcli-stub.sh" bash "$AF" --workers 8 --facts-dir "$P/facts" "$@" ) > "$W/out" 2>&1 \ + || { echo "app-facts failed:"; tail -5 "$W/out"; exit 2; } + DESC=$(grep -c . "$DESCRIBE_LOG"); SECS=$(( $(date +%s) - t0 )) +} + +echo "bench-app-sweep: $N units, all loop-containing; $CHANGED changed ($PCT %)" +printf '%-34s %10s %8s\n' "refresh" "describes" "wall s" +build 1; run; printf '%-34s %10s %8s\n' "first map (full)" "$DESC" "$SECS"; D_FIRST=$DESC +run; printf '%-34s %10s %8s\n' "refresh, nothing changed" "$DESC" "$SECS"; D_NONE=$DESC +build 2; run; printf '%-34s %10s %8s\n' "refresh, $PCT % changed (incremental)" "$DESC" "$SECS"; D_INC=$DESC +run --full; printf '%-34s %10s %8s\n' "refresh, $PCT % changed (--full)" "$DESC" "$SECS"; D_FULL=$DESC +echo "(wall time is the stub's; on a real model multiply describes by the per-describe cost)" +if [ "$D_FIRST" -eq "$N" ] && [ "$D_NONE" -eq 0 ] && [ "$D_INC" -eq "$CHANGED" ] && [ "$D_FULL" -eq "$N" ]; then + echo "OK: incremental describes $D_INC of $N ($PCT %), full describes $D_FULL" + exit 0 +fi +echo "FAIL: expected $N / 0 / $CHANGED / $N describes, got $D_FIRST / $D_NONE / $D_INC / $D_FULL" +exit 1 diff --git a/tests/wave2/fixtures/app-facts-incremental/mkmodel.py b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py new file mode 100755 index 00000000..c549a2aa --- /dev/null +++ b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +# mkmodel.py -- build a synthetic MPR v2 model, its prepared catalog and the stub's describe +# bodies, for tests/wave2/test-app-facts-incremental.sh and tests/wave2/bench-app-sweep.sh. +# +# python3 mkmodel.py [] +# +# spec.tsv, one microflow per line: +# All microflows live in MyFirstModule. changes the unit's bytes (so its ContentsHash) +# and the MDL the stub mxcli "describes" for it; moves the unit without changing them. +# +# WHAT IS CAPTURED AND WHAT IS NOT. With (tests/wave2/fixtures/ +# mx-metadata-note), the module unit and the MyFirstLogic microflow are the real .mxunit bytes and +# real Unit-table container of a Mendix 11.15.0 model, so the UnitID -> qualified-name mapping is +# exercised on real BSON. Every other unit is synthetic BSON carrying only $Type, Name and +# Documentation. ContentsHash is base64(sha256()), the relation measured on all 2,337 +# units of a field app's Unit table. The Unit table schema is the captured one, verbatim. +import base64, hashlib, os, shutil, sqlite3, struct, sys + +md, prep, spec = sys.argv[1:4] +cap = sys.argv[4] if len(sys.argv) > 4 else None +MODNAME = "MyFirstModule" +MOD = "16747dc6-b6b7-42ae-aabf-255dca2aeeaf" +ROOT = bytes.fromhex("4d827743a25b514b800e658754fc369b") # the captured project root container +CAPTURED_MF = {"MyFirstLogic": "4a8475b3-14cb-4100-8963-58f2fd3a0be8"} +SCHEMA = """CREATE TABLE _MetaData (_FormatVersion INTEGER, _ProductVersion TEXT, _BuildVersion TEXT, _SchemaHash TEXT); +CREATE TABLE Unit + ( + UnitID BLOB PRIMARY KEY NOT NULL, + ContainerID BLOB, + ContainmentName TEXT, + TreeConflict LONG, + ContentsHash TEXT, + ContentsConflicts TEXT + );""" + + +def blob(g): + b = bytes.fromhex(g.replace("-", "")) + return b[3::-1] + b[5:3:-1] + b[7:5:-1] + b[8:] + + +def uid(seed): + h = hashlib.md5(seed.encode()).hexdigest() + h = h[:12] + "4" + h[13:16] + "8" + h[17:] + return f"{h[:8]}-{h[8:12]}-{h[12:16]}-{h[16:20]}-{h[20:32]}" + + +def bson(fields): + body = b"".join(b"\x02" + k.encode() + b"\x00" + struct.pack("`: a function of the unit's version, nothing else. +bodies = os.path.join(md, "stub-bodies") +shutil.rmtree(bodies, ignore_errors=True); os.makedirs(bodies) +for name, ver, _, loops in flows: + qn = MODNAME + "." + name + inner = (" loop $Item in $List\n retrieve $Line from %s.Line;\n log info node 'v%s' 'step';\n end loop;\n" + % (MODNAME, ver)) if loops else " log info node 'v%s' 'no loop';\n" % ver + open(os.path.join(bodies, qn + ".mdl"), "w").write( + "create or modify microflow %s ()\nbegin\n%send;\n/\n" % (qn, inner)) + +# The prepared catalog the stub's graph-report drops in place: every table app-facts reads. +if os.path.exists(prep): + os.unlink(prep) +c = sqlite3.connect(prep) +c.executescript(""" +CREATE TABLE catalog_meta (Key TEXT, Value TEXT); +CREATE TABLE modules_data (Name TEXT, Source TEXT, AppStoreVersion TEXT, AppStoreGuid TEXT); +CREATE TABLE entities_data (QualifiedName TEXT, ModuleName TEXT, EntityType TEXT, AttributeCount INTEGER); +CREATE TABLE microflows_data (QualifiedName TEXT, ModuleName TEXT, Name TEXT, ActivityCount INTEGER, MicroflowType TEXT); +CREATE TABLE activities_data (MicroflowQualifiedName TEXT, ModuleName TEXT, ActivityType TEXT); +CREATE TABLE pages_data (QualifiedName TEXT, ModuleName TEXT); +CREATE TABLE scheduled_events_data (QualifiedName TEXT, ModuleName TEXT, Microflow TEXT, Enabled TEXT, IntervalSeconds INTEGER, RepeatDescription TEXT); +CREATE TABLE refs (SourceType TEXT, SourceName TEXT, TargetName TEXT, RefKind TEXT, ModuleName TEXT); +CREATE TABLE objects (QualifiedName TEXT, ModuleName TEXT, ObjectType TEXT); +CREATE TABLE projects (ProjectName TEXT, MendixVersion TEXT); +CREATE TABLE graph_dead_assets (QualifiedName TEXT, ObjectType TEXT, ModuleName TEXT); +INSERT INTO catalog_meta VALUES ('build_mode','full'); +INSERT INTO modules_data VALUES ('MyFirstModule','','',''); +INSERT INTO projects VALUES ('App','11.15.0'); +INSERT INTO entities_data VALUES ('MyFirstModule.Line','MyFirstModule','PERSISTENT',3); +""") +names = [MODNAME + "." + n for n, _, _, _ in flows] +for (n, _, _, loops), qn in zip(flows, names): + c.execute("INSERT INTO microflows_data VALUES (?,?,?,?,?)", (qn, MODNAME, n, 4, "MICROFLOW")) + c.execute("INSERT INTO objects VALUES (?,?,?)", (qn, MODNAME, "MICROFLOW")) + c.execute("INSERT INTO activities_data VALUES (?,?,?)", (qn, MODNAME, "ActionActivity")) + if loops: + c.execute("INSERT INTO activities_data VALUES (?,?,?)", (qn, MODNAME, "LoopedActivity")) +for a, b in zip(names, names[1:]): + c.execute("INSERT INTO refs VALUES ('MICROFLOW',?,?,'call',?)", (a, b, MODNAME)) +c.commit(); c.close() diff --git a/tests/wave2/fixtures/app-facts-incremental/mxcli-stub.sh b/tests/wave2/fixtures/app-facts-incremental/mxcli-stub.sh new file mode 100755 index 00000000..0d573685 --- /dev/null +++ b/tests/wave2/fixtures/app-facts-incremental/mxcli-stub.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# A stub mxcli for the app-facts incremental-sweep fixture and bench. It never reads the model: +# --version prints $STUB_VERSION (a change forces a full sweep) +# graph-report copies $PREP_DB to .mxcli/catalog.db and writes an empty report to -o +# describe appends to $DESCRIBE_LOG and prints stub-bodies/.mdl; +# exits 1 for any name listed in $STUB_FAIL (space-separated) +case "${1:-}" in + --version) echo "${STUB_VERSION:-mxcli stub 1.0}" ;; + graph-report) + o=""; while [ $# -gt 0 ]; do [ "$1" = -o ] && o="$2"; shift; done + mkdir -p .mxcli && cp "$PREP_DB" .mxcli/catalog.db && echo '{}' > "$o" ;; + describe) + echo "$2" >> "$DESCRIBE_LOG" + case " ${STUB_FAIL:-} " in *" $2 "*) exit 1 ;; esac + cat "stub-bodies/$2.mdl" ;; + *) exit 1 ;; +esac diff --git a/tests/wave2/test-app-facts-incremental.sh b/tests/wave2/test-app-facts-incremental.sh new file mode 100755 index 00000000..b1353463 --- /dev/null +++ b/tests/wave2/test-app-facts-incremental.sh @@ -0,0 +1,242 @@ +#!/usr/bin/env bash +# test-app-facts-incremental.sh — the incremental loop sweep in project-bin/app-facts.sh and the +# "app map" freshness line in bin/status.sh (#297). +# +# Usage: bash tests/wave2/test-app-facts-incremental.sh [path-to-app-facts.sh] +# (bin/status.sh and mx-metadata-note.py are taken from the same toolkit as the subject) +# +# The defect: every app-facts run re-described every loop-containing microflow. A field run on a +# 17,500-unit app took 2 h 45 at 6 workers for 1,917 describes, while ~80 units had changed +# since the previous map. The fix keys each mdl/ output on its unit's ContentsHash (the .mpr Unit +# table) and module/folder path, and re-describes only what changed. +# +# Model, catalog and mxcli are built by fixtures/app-facts-incremental/: the module and one +# microflow are REAL captured units (fixtures/mx-metadata-note, Mendix 11.15.0) and the rest is +# synthetic BSON; the stub mxcli counts describes in a log. See mkmodel.py for what is captured. +# +# Cases, each on BOTH layouts (single-tree, and two-tree with the .mpr under app/): +# T1 first run describes all candidates (full: no index yet); snapshot + index in manifest +# T2 second run, nothing changed: 0 describes, every output kept +# T3 one unit changed: exactly 1 describe, its mdl/ file rewritten +# T3b a stale Unit-table ContentsHash (file changed, row not) still triggers that 1 describe +# T4 one unit removed: 0 describes, its mdl/ file and loops.json entry gone +# T5 a unit moved to a folder (hash unchanged, path changed): 1 describe +# T6 a describe that fails is not indexed, so the next run retries it +# T7 incremental outputs are byte-identical to a --full run and to a fresh full run +# T8 --full re-describes everything; a changed mxcli version forces a full sweep +# T9 a model whose Unit table cannot be read: full sweep, a reason, never a false incremental +# T10 --freshness: "fresh", then "N units changed since "; silent with no map +# T11 bin/status.sh shows the app map line (full screen and --brief); no line without a map +# T12 the real captured units map to MyFirstModule.MyFirstLogic through mx-metadata-note.py units + +set -uo pipefail +TK="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +AF="${1:-$TK/project-bin/app-facts.sh}" +case "$AF" in /*) ;; *) AF="$PWD/$AF" ;; esac +SRC="$(cd "$(dirname "$AF")/.." && pwd)" +STATUS="$SRC/bin/status.sh" +FX="$TK/tests/wave2/fixtures/app-facts-incremental" +CAP="$TK/tests/wave2/fixtures/mx-metadata-note" +command -v sqlite3 >/dev/null || { echo "SKIP: sqlite3 not on PATH"; echo "PASS=0 FAIL=0"; exit 0; } +. "$TK/bin/lib/portable.sh"; require_py + +W="$(mktemp -d "${TMPDIR:-/tmp}/af-incr.XXXXXX")" || exit 2 +trap 'rm -rf "$W"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); printf ' ok %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; [ -n "${2:-}" ] && printf '%s\n' "$2" | sed 's/^/ /'; } +check() { if [ "$2" = "$3" ]; then ok "$1"; else bad "$1" "expected '$3', got '$2'"; fi; } +jq_() { "$PY" -c 'import json,sys +d=json.load(open(sys.argv[1])) +for k in sys.argv[2].split("/"): + d = d.get(k, "MISSING") if isinstance(d, dict) else "MISSING" +print(json.dumps(d) if isinstance(d,(dict,list)) else d)' "$1" "$2"; } + +export STUB_VERSION="mxcli stub 1.0" +M=MyFirstModule +SPEC_BASE="MyFirstLogic 1 - 1 +SUB_A 1 - 1 +SUB_B 1 - 1 +SUB_C 1 - 1 +SUB_D 1 - 1 +ACT_NoLoop 1 - 0" + +# model : (re)build the model under $MD and the prepared catalog $PREP_DB +model() { printf '%s\n' "$1" > "$W/spec.tsv"; "$PY" "$FX/mkmodel.py" "$MD" "$PREP_DB" "$W/spec.tsv" "$CAP" || bad "mkmodel failed"; } +# run [args] : one app-facts run; describes counted from the stub's log +run() { + : > "$DESCRIBE_LOG" + ( cd "$P" && PROJECT_ROOT="$P" MXCLI="$FX/mxcli-stub.sh" PREP_DB="$PREP_DB" DESCRIBE_LOG="$DESCRIBE_LOG" \ + bash "$AF" --workers 3 --facts-dir "$FACTS" "$@" ) > "$W/out" 2>&1 + RC=$? + NDESC=$(grep -c . "$DESCRIBE_LOG") +} +mdls() { ls "$FACTS/mdl" 2>/dev/null | tr '\n' ' '; } + +for layout in single two; do + echo "== $layout-tree layout" + P="$W/$layout"; mkdir -p "$P" + if [ "$layout" = two ]; then MD="$P/app"; else MD="$P"; fi + PREP_DB="$W/$layout-prep.db"; DESCRIBE_LOG="$W/$layout-describe.log"; FACTS="$P/analysis/app-facts" + export PREP_DB DESCRIBE_LOG + + # --- T1 first run -------------------------------------------------------------------------- + model "$SPEC_BASE" + run + check "[$layout] T1 first run exits 0" "$RC" "0" + [ "$RC" = 0 ] || { tail -5 "$W/out"; continue; } + check "[$layout] T1 first run describes all 5 loop microflows" "$NDESC" "5" + check "[$layout] T1 sweep mode full (no index yet)" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/mode)" "full" + check "[$layout] T1 snapshot holds every unit (module + 6 microflows)" "$(jq_ "$FACTS/manifest.json" units/count)" "7" + check "[$layout] T1 index holds the 5 described outputs" \ + "$("$PY" -c 'import json,sys;print(len(json.load(open(sys.argv[1]))["mdl_index"]))' "$FACTS/manifest.json")" "5" + check "[$layout] T1 the real captured microflow is indexed by its real UnitID" \ + "$(jq_ "$FACTS/manifest.json" "mdl_index/$M.MyFirstLogic/unit")" "4a8475b3-14cb-4100-8963-58f2fd3a0be8" + for t in .units.tsv .sweep-plan.json .describe-todo.txt; do + [ -e "$FACTS/$t" ] && bad "[$layout] T1 temp file $t left behind" + done + + # --- T2 nothing changed ---------------------------------------------------------------------- + B_MTIME="$(ls -l "$FACTS/mdl/$M.SUB_B.mdl")" + run + check "[$layout] T2 unchanged model: 0 describes" "$NDESC" "0" + check "[$layout] T2 sweep mode incremental" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/mode)" "incremental" + check "[$layout] T2 all 5 outputs reused" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reused)" "5" + check "[$layout] T2 loops section still counts 5 described files" "$(jq_ "$FACTS/manifest.json" sections/loops/described)" "5" + check "[$layout] T2 0 units changed since the previous map" "$(jq_ "$FACTS/manifest.json" units/changed_since_previous_map/changed)" "0" + + # --- T3 one unit changed ---------------------------------------------------------------------- + model "$(printf '%s\n' "$SPEC_BASE" | sed 's/^SUB_B 1/SUB_B 2/')" + run + check "[$layout] T3 one changed unit: exactly 1 describe" "$NDESC" "1" + check "[$layout] T3 the describe was the changed unit" "$(cat "$DESCRIBE_LOG")" "$M.SUB_B" + grep -q "'v2'" "$FACTS/mdl/$M.SUB_B.mdl" && ok "[$layout] T3 its mdl/ file holds the new version" \ + || bad "[$layout] T3 mdl/ file for the changed unit is stale" + check "[$layout] T3 manifest counts 1 changed unit" "$(jq_ "$FACTS/manifest.json" units/changed_since_previous_map/changed)" "1" + + # --- T3b a stale Unit-table ContentsHash does not hide a changed unit ------------------------ + # Measured on a field app's history: at one committed state one unit's column disagreed with + # its file. Here SUB_B's file changes while its row keeps the hash the last map recorded. + model "$(printf '%s\n' "$SPEC_BASE" | sed 's/^SUB_B 1/SUB_B 3/')" + "$PY" - "$MD/App.mpr" "$FACTS/manifest.json" "$M.SUB_B" <<'PY' +import json, sqlite3, sys +man = json.load(open(sys.argv[2])); u = man["mdl_index"][sys.argv[3]]["unit"] +b = bytes.fromhex(u.replace("-", "")); b = b[3::-1] + b[5:3:-1] + b[7:5:-1] + b[8:] +db = sqlite3.connect(sys.argv[1]); db.execute("update Unit set ContentsHash=? where UnitID=?", (man["units"]["snapshot"][u], b)); db.commit() +PY + run + check "[$layout] T3b stale table hash, changed file: still 1 describe" "$(cat "$DESCRIBE_LOG")" "$M.SUB_B" + case "$(jq_ "$FACTS/manifest.json" units/reason)" in *"1 unit(s) whose Unit-table ContentsHash disagrees"*) + ok "[$layout] T3b the stale row is counted in the manifest" ;; + *) bad "[$layout] T3b stale row not reported" "$(jq_ "$FACTS/manifest.json" units/reason)" ;; esac + model "$(printf '%s\n' "$SPEC_BASE" | sed 's/^SUB_B 1/SUB_B 2/')"; run + check "[$layout] T3b back to v2: 1 describe" "$NDESC" "1" + + # --- T4 one unit removed ---------------------------------------------------------------------- + SPEC_NOW="$(printf '%s\n' "$SPEC_BASE" | sed 's/^SUB_B 1/SUB_B 2/' | grep -v '^SUB_C ')" + model "$SPEC_NOW" + run + check "[$layout] T4 removed unit: 0 describes" "$NDESC" "0" + [ ! -e "$FACTS/mdl/$M.SUB_C.mdl" ] && ok "[$layout] T4 the removed unit's mdl/ file is deleted" \ + || bad "[$layout] T4 stale mdl/ file kept for a removed unit" "$(mdls)" + check "[$layout] T4 loops.json no longer lists it" "$(jq_ "$FACTS/loops.json" "microflows/$M.SUB_C")" "MISSING" + check "[$layout] T4 removed_outputs = 1" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/removed_outputs)" "1" + check "[$layout] T4 manifest counts 1 removed unit" "$(jq_ "$FACTS/manifest.json" units/changed_since_previous_map/removed)" "1" + + # --- T5 moved to a folder: same bytes, new path --------------------------------------------- + SPEC_NOW="$(printf '%s\n' "$SPEC_NOW" | sed 's/^SUB_A 1 -/SUB_A 1 Batch/')" + model "$SPEC_NOW" + run + check "[$layout] T5 moved unit (hash unchanged, folder changed): 1 describe" "$(cat "$DESCRIBE_LOG")" "$M.SUB_A" + check "[$layout] T5 index records the folder path" "$(jq_ "$FACTS/manifest.json" "mdl_index/$M.SUB_A/path")" "$M/Batch/SUB_A" + + # --- T6 a failed describe is retried --------------------------------------------------------- + SPEC_NOW="$(printf '%s\n' "$SPEC_NOW" | sed 's/^SUB_D 1/SUB_D 2/')" + model "$SPEC_NOW" + STUB_FAIL="$M.SUB_D" run + check "[$layout] T6 failing describe: loops status partial" "$(jq_ "$FACTS/manifest.json" sections/loops/status)" "partial" + check "[$layout] T6 the failed unit is not indexed" "$(jq_ "$FACTS/manifest.json" "mdl_index/$M.SUB_D")" "MISSING" + [ ! -e "$FACTS/mdl/$M.SUB_D.mdl" ] && ok "[$layout] T6 no stale mdl/ file for the failed unit" \ + || bad "[$layout] T6 the old SUB_D output survived a failed re-describe" + run + check "[$layout] T6 next run retries exactly the failed unit" "$(cat "$DESCRIBE_LOG")" "$M.SUB_D" + check "[$layout] T6 loops status back to pass" "$(jq_ "$FACTS/manifest.json" sections/loops/status)" "pass" + + # --- T7 incremental == full ------------------------------------------------------------------- + INC="$W/$layout-inc"; rm -rf "$INC"; cp -R "$FACTS" "$INC" + FACTS_SAVE="$FACTS"; FACTS="$W/$layout-fresh"; rm -rf "$FACTS" + run + check "[$layout] T7 fresh full run describes the 4 current candidates" "$NDESC" "4" + same=1 + for f in loops.json inventory.json dependencies.json loops-expected.json loops-candidates.tsv loops-scheduled.json; do + cmp -s "$INC/$f" "$FACTS/$f" || { same=0; bad "[$layout] T7 $f differs between incremental and full"; } + done + diff -r "$INC/mdl" "$FACTS/mdl" > "$W/mdl.diff" 2>&1 || { same=0; bad "[$layout] T7 mdl/ differs" "$(head -5 "$W/mdl.diff")"; } + [ "$same" = 1 ] && ok "[$layout] T7 incremental outputs byte-identical to a fresh full run (mdl/ + 6 fact files)" + FACTS="$FACTS_SAVE" + + # --- T8 --full and an mxcli upgrade ---------------------------------------------------------- + run --full + check "[$layout] T8 --full re-describes all 4" "$NDESC" "4" + check "[$layout] T8 --full recorded as the reason" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reason)" "--full" + diff -r "$INC/mdl" "$FACTS/mdl" >/dev/null 2>&1 && cmp -s "$INC/loops.json" "$FACTS/loops.json" \ + && ok "[$layout] T8 --full in place gives the same outputs as the incremental run" \ + || bad "[$layout] T8 --full in place differs from the incremental outputs" + STUB_VERSION="mxcli stub 2.0" run + check "[$layout] T8 mxcli version change: full sweep of all 4" "$NDESC" "4" + case "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reason)" in + "mxcli changed"*) ok "[$layout] T8 reason names the mxcli change" ;; + *) bad "[$layout] T8 reason does not name the mxcli change" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reason)" ;; + esac + + # --- T10 freshness (before T9 breaks the model) ---------------------------------------------- + fresh() { ( cd "$P" && PROJECT_ROOT="$P" bash "$AF" --freshness --facts-dir "$FACTS" ) 2>&1; } + case "$(fresh)" in "app map: fresh (built "*) ok "[$layout] T10 freshness says fresh right after a map" ;; + *) bad "[$layout] T10 freshness after a map" "$(fresh)" ;; esac + model "$(printf '%s\n' "$SPEC_NOW" | sed 's/^SUB_A 1/SUB_A 3/; s/^MyFirstLogic 1/MyFirstLogic 1/') +NEW_One 1 - 1" + case "$(fresh)" in "app map: 2 units changed since "2[0-9][0-9][0-9]-*) ok "[$layout] T10 one changed + one added unit: '2 units changed since '" ;; + *) bad "[$layout] T10 changed-unit count" "$(fresh)" ;; esac + check "[$layout] T10 no map: silent" "$( (cd "$P" && PROJECT_ROOT="$P" bash "$AF" --freshness --facts-dir "$W/nowhere") 2>&1)" "" + + # --- T11 status.sh ----------------------------------------------------------------------------- + printf '# PROJECT\n\n## Current stage\n**Stage 1 in progress.**\n' > "$P/PROJECT.md" + if [ "$layout" = single ]; then + S_FULL="$(bash "$STATUS" "$P" 2>/dev/null)"; S_BRIEF="$(bash "$STATUS" "$P" --brief 2>/dev/null)" + printf '%s\n' "$S_FULL" | grep -q "app map: 2 units changed since " && ok "[$layout] T11 status.sh full screen shows the app map line" \ + || bad "[$layout] T11 status.sh full screen lacks the app map line" "$(printf '%s\n' "$S_FULL" | head -8)" + printf '%s\n' "$S_BRIEF" | grep '^STATE' | grep -q "app map: 2 units changed since " && ok "[$layout] T11 status.sh --brief STATE carries it" \ + || bad "[$layout] T11 status.sh --brief lacks the app map" "$S_BRIEF" + mv "$FACTS/manifest.json" "$W/man.bak" + bash "$STATUS" "$P" 2>/dev/null | grep -q "app map" && bad "[$layout] T11 status.sh printed an app map line with no map" \ + || ok "[$layout] T11 no map, no app map line" + mv "$W/man.bak" "$FACTS/manifest.json" + fi + + # --- T9 unreadable Unit table ---------------------------------------------------------------- + P9="$W/$layout-v1"; mkdir -p "$P9"; if [ "$layout" = two ]; then MD9="$P9/app"; else MD9="$P9"; fi + MD_SAVE="$MD"; P_SAVE="$P"; FACTS_SAVE="$FACTS" + MD="$MD9"; P="$P9"; FACTS="$P9/analysis/app-facts" + model "$SPEC_BASE"; : > "$MD/App.mpr" # an .mpr with no Unit table + run; run + check "[$layout] T9 no Unit table: second run is still a full sweep (5 describes)" "$NDESC" "5" + check "[$layout] T9 units section is a fault, not absent" "$(jq_ "$FACTS/manifest.json" units/status)" "fault" + case "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reason)" in "no unit snapshot"*) ok "[$layout] T9 reason: no unit snapshot" ;; + *) bad "[$layout] T9 reason" "$(jq_ "$FACTS/manifest.json" sections/loops/sweep/reason)" ;; esac + MD="$MD_SAVE"; P="$P_SAVE"; FACTS="$FACTS_SAVE" +done + +# --- T12 the real captured units, through the shared mapping --------------------------------- +echo "== T12 mx-metadata-note.py units on captured units" +MD="$W/t12"; PREP_DB="$W/t12.db"; model "MyFirstLogic 1 - 1" +L="$("$PY" "$SRC/project-bin/mx-metadata-note.py" units "$MD/App.mpr" | grep '^4a8475b3')" +check "T12 real microflow unit -> type, qualified name, path" "$(printf '%s\n' "$L" | cut -f3-5)" \ + "Microflows\$Microflow MyFirstModule.MyFirstLogic MyFirstModule/MyFirstLogic" +check "T12 hash is the Unit table's ContentsHash = base64(sha256(.mxunit))" "$(printf '%s\n' "$L" | cut -f2)" \ + "$("$PY" -c 'import base64,hashlib,sys;print(base64.b64encode(hashlib.sha256(open(sys.argv[1],"rb").read()).digest()).decode())' "$CAP/4a8475b3-14cb-4100-8963-58f2fd3a0be8.mxunit")" +check "T12 the module unit itself has no qualified name" \ + "$("$PY" "$SRC/project-bin/mx-metadata-note.py" units "$MD/App.mpr" | grep '^16747dc6' | cut -f3,4)" "Projects\$ModuleImpl " + +echo "" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ] From 6550815effe3a101a1573874028081cb683c8877 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:28:30 +0000 Subject: [PATCH 2/3] fix(app-facts): drop figures taken from an out-of-scope repository (#297) Remove the unit counts, timings and stale-row observation that came from a project outside this repository. The bytes-over-column hash rule stays, justified on its own terms. Field evidence is now: no field run; checked on synthetic and captured fixtures only. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 2 +- project-bin/app-facts.sh | 4 ++-- project-bin/mx-metadata-note.py | 6 +++--- tests/wave2/fixtures/app-facts-incremental/mkmodel.py | 4 ++-- tests/wave2/test-app-facts-incremental.sh | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 578e16da..2d15e0e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased -- new(app-facts): **the app map's loop sweep is incremental, and `bin/status.sh` says how stale the map is** ([#297](https://github.com/MendixMau/mxcli-project-toolkit/issues/297)). Every run re-described every loop-containing microflow: on a 17,500-unit app that was 1,917 describes in 2 h 45 at 6 workers (about 31 s each, not the 1-2 s `app-analysis.md` promised), while about 80 units had changed between rounds. `manifest.json` now stores a `{UnitID: ContentsHash}` snapshot of the `.mpr` Unit table and an `mdl_index` recording the unit state each `mdl/` file was described from. A refresh describes only candidates whose hash, module or folder changed, or that have no output, keeps the rest and deletes outputs of removed or loop-free units. The sweep is full on the first run, with `--full`, after an mxcli version change, or when the Unit table cannot be read (MPR v1); `sections.loops.sweep` says which and why. UnitID-to-name mapping is `mx-metadata-note.py units`, the Studio Pro History note's own walk, refactored into a shared `Model` base for git revisions and the working tree. The hash is taken from the `.mxunit` bytes in the column's format: on a field app's history the column matched the file for all 2,337 units at HEAD, but one row was stale at 1 of 8 committed states, which would have hidden a change. `bin/status.sh` prints `app map: N units changed since ` or `app map: fresh` when a map exists (`app-facts.sh --freshness`, no mxcli needed), and nothing otherwise. `app-analysis.md` gives the measured cost by app size. Fixture `tests/wave2/test-app-facts-incremental.sh`, 92 checks on both layouts with captured template units and a stub mxcli: all describes on a first run, 0 when nothing changed, 1 for a changed, moved, or stale-row unit, a removed unit's output deleted, a failed describe retried, outputs byte-identical to `--full` and to a fresh full run, and the status line; 66 checks fail against master. `tests/wave2/bench-app-sweep.sh`, 2,000 units with 1 % changed: 20 describes incremental against 2,000 full. Field evidence: the unit mapping ran on a 2,337-unit Mendix app (876 microflows named, no duplicates, 1.2 s); no run against a real mxcli describe yet. Not built: replacing describe with catalog queries (mxcli v0.25 `activities` with loop depth), NEEDS-FIELD-DATA, a real v0.25 catalog capture to check parity with the parser. — jopterhorst (#297) +- new(app-facts): **the app map's loop sweep is incremental, and `bin/status.sh` says how stale the map is** ([#297](https://github.com/MendixMau/mxcli-project-toolkit/issues/297)). Every run re-described every loop-containing microflow: on a 17,500-unit app that was 1,917 describes in 2 h 45 at 6 workers (about 31 s each, not the 1-2 s `app-analysis.md` promised), while about 80 units had changed between rounds. `manifest.json` now stores a `{UnitID: ContentsHash}` snapshot of the `.mpr` Unit table and an `mdl_index` recording the unit state each `mdl/` file was described from. A refresh describes only candidates whose hash, module or folder changed, or that have no output, keeps the rest and deletes outputs of removed or loop-free units. The sweep is full on the first run, with `--full`, after an mxcli version change, or when the Unit table cannot be read (MPR v1); `sections.loops.sweep` says which and why. UnitID-to-name mapping is `mx-metadata-note.py units`, the Studio Pro History note's own walk, refactored into a shared `Model` base for git revisions and the working tree. The hash is computed from the `.mxunit` bytes in the column's format, so a Unit-table row that lags its file cannot hide a change; disagreements are counted in the manifest. `bin/status.sh` prints `app map: N units changed since ` or `app map: fresh` when a map exists (`app-facts.sh --freshness`, no mxcli needed), and nothing otherwise. `app-analysis.md` gives the measured cost by app size. Fixture `tests/wave2/test-app-facts-incremental.sh`, 92 checks on both layouts with captured template units and a stub mxcli: all describes on a first run, 0 when nothing changed, 1 for a changed, moved, or stale-row unit, a removed unit's output deleted, a failed describe retried, outputs byte-identical to `--full` and to a fresh full run, and the status line; 66 checks fail against master. `tests/wave2/bench-app-sweep.sh`, 2,000 units with 1 % changed: 20 describes incremental against 2,000 full. Field evidence: no field run; checked on synthetic and captured fixtures only. Not built: replacing describe with catalog queries (mxcli v0.25 `activities` with loop depth), NEEDS-FIELD-DATA, a real v0.25 catalog capture to check parity with the parser. — jopterhorst (#297) - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/project-bin/app-facts.sh b/project-bin/app-facts.sh index 167febfc..94131e18 100755 --- a/project-bin/app-facts.sh +++ b/project-bin/app-facts.sh @@ -26,7 +26,7 @@ # # INCREMENTAL SWEEP (#297). manifest.json keeps a {UnitID: ContentsHash} snapshot of the .mpr # Unit table (the hash taken from the .mxunit bytes, which the column normally equals; see -# mx-metadata-note.py units for the one stale row measured), and an index of which unit state each mdl/ file was described from. A refresh +# mx-metadata-note.py units for why the bytes win), and an index of which unit state each mdl/ file was described from. A refresh # re-describes only a candidate whose unit is new, changed (hash), moved (module/folder path) # or has no output yet; it keeps every other mdl/ file and deletes the outputs of units that are # gone or no longer loop. UnitID -> name comes from mx-metadata-note.py (beside this script): @@ -563,7 +563,7 @@ T2=$(( $(date +%s) - T2 )) # --------------------------------------------------------------------------------------------- # Step 2b — unit snapshot: {UnitID: ContentsHash} from the .mpr Unit table, with each unit's -# type, qualified name and folder path. Seconds, not minutes (0.3 s on 2,337 units). A model +# type, qualified name and folder path. One read of each .mxunit file, no mxcli. A model # whose table cannot be read gets a full sweep, never a guessed incremental one. # --------------------------------------------------------------------------------------------- UNITS_TSV="$FACTS_DIR/.units.tsv"; UNITS_STATUS="fault"; UNITS_REASON="" diff --git a/project-bin/mx-metadata-note.py b/project-bin/mx-metadata-note.py index 452cc0f2..91615e09 100755 --- a/project-bin/mx-metadata-note.py +++ b/project-bin/mx-metadata-note.py @@ -224,9 +224,9 @@ def units(mpr, hashes_only=False): folder path (Module/Folder/.../Name). The hash has the Unit table's ContentsHash format, base64(sha256()), and - is computed from the bytes. Measured on a field app's history: the column equals that value on - all 2,337 units at HEAD, Studio Pro- and mxcli-written alike, but at one of 8 committed states - one unit's column was stale (its file had changed, its row had not). The bytes are what + is computed from the bytes. The column normally holds that same value, but nothing guarantees + a row is rewritten in the same moment as its file (a save in progress, a checkout of the two + at different states), and a stale row would hide a change. The bytes are what `mxcli describe` reads, so they win; the column is the fallback for a unit with no file, and each disagreement is counted on stderr.""" t = Tree(mpr) diff --git a/tests/wave2/fixtures/app-facts-incremental/mkmodel.py b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py index c549a2aa..180e4bc8 100755 --- a/tests/wave2/fixtures/app-facts-incremental/mkmodel.py +++ b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py @@ -12,8 +12,8 @@ # mx-metadata-note), the module unit and the MyFirstLogic microflow are the real .mxunit bytes and # real Unit-table container of a Mendix 11.15.0 model, so the UnitID -> qualified-name mapping is # exercised on real BSON. Every other unit is synthetic BSON carrying only $Type, Name and -# Documentation. ContentsHash is base64(sha256()), the relation measured on all 2,337 -# units of a field app's Unit table. The Unit table schema is the captured one, verbatim. +# Documentation. ContentsHash is base64(sha256()), the format mx-metadata-note.py units +# computes. The Unit table schema is the captured one, verbatim. import base64, hashlib, os, shutil, sqlite3, struct, sys md, prep, spec = sys.argv[1:4] diff --git a/tests/wave2/test-app-facts-incremental.sh b/tests/wave2/test-app-facts-incremental.sh index b1353463..111c0442 100755 --- a/tests/wave2/test-app-facts-incremental.sh +++ b/tests/wave2/test-app-facts-incremental.sh @@ -115,8 +115,8 @@ for layout in single two; do check "[$layout] T3 manifest counts 1 changed unit" "$(jq_ "$FACTS/manifest.json" units/changed_since_previous_map/changed)" "1" # --- T3b a stale Unit-table ContentsHash does not hide a changed unit ------------------------ - # Measured on a field app's history: at one committed state one unit's column disagreed with - # its file. Here SUB_B's file changes while its row keeps the hash the last map recorded. + # Nothing guarantees a row is rewritten together with its file. Here SUB_B's file changes while + # its row keeps the hash the last map recorded. model "$(printf '%s\n' "$SPEC_BASE" | sed 's/^SUB_B 1/SUB_B 3/')" "$PY" - "$MD/App.mpr" "$FACTS/manifest.json" "$M.SUB_B" <<'PY' import json, sqlite3, sys From e52e778f3b244acd0ffefa1ed72b22164b6e4a65 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:49:20 +0000 Subject: [PATCH 3/3] fix(app-facts): review fixes for the incremental sweep (#297) - --freshness passed the whole unit hash list to Python in one env var; Linux caps that at 128 KB (~1,600 units), so on large apps it failed with 'Argument list too long' and status.sh silently lost the line. The list now goes through a temp file, cleaned up on exit. - A --skip-loops run keeps the last loop sweep's snapshot and its date (units.snapshot_taken), so a stale mdl/ never reads fresh. - skills/app-analysis.md: --full after edits made outside Studio Pro. - Fixture +16 checks: --freshness and status.sh on 2,101 units (fails on 6550815), module rename, --skip-loops freshness. Bench calls --freshness. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 2 +- project-bin/app-facts.sh | 42 +++++++++++++------ skills/app-analysis.md | 2 + tests/wave2/bench-app-sweep.sh | 15 ++++++- .../fixtures/app-facts-incremental/mkmodel.py | 6 +-- tests/wave2/test-app-facts-incremental.sh | 41 ++++++++++++++++++ 6 files changed, 90 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d15e0e9..6d6ddb54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased -- new(app-facts): **the app map's loop sweep is incremental, and `bin/status.sh` says how stale the map is** ([#297](https://github.com/MendixMau/mxcli-project-toolkit/issues/297)). Every run re-described every loop-containing microflow: on a 17,500-unit app that was 1,917 describes in 2 h 45 at 6 workers (about 31 s each, not the 1-2 s `app-analysis.md` promised), while about 80 units had changed between rounds. `manifest.json` now stores a `{UnitID: ContentsHash}` snapshot of the `.mpr` Unit table and an `mdl_index` recording the unit state each `mdl/` file was described from. A refresh describes only candidates whose hash, module or folder changed, or that have no output, keeps the rest and deletes outputs of removed or loop-free units. The sweep is full on the first run, with `--full`, after an mxcli version change, or when the Unit table cannot be read (MPR v1); `sections.loops.sweep` says which and why. UnitID-to-name mapping is `mx-metadata-note.py units`, the Studio Pro History note's own walk, refactored into a shared `Model` base for git revisions and the working tree. The hash is computed from the `.mxunit` bytes in the column's format, so a Unit-table row that lags its file cannot hide a change; disagreements are counted in the manifest. `bin/status.sh` prints `app map: N units changed since ` or `app map: fresh` when a map exists (`app-facts.sh --freshness`, no mxcli needed), and nothing otherwise. `app-analysis.md` gives the measured cost by app size. Fixture `tests/wave2/test-app-facts-incremental.sh`, 92 checks on both layouts with captured template units and a stub mxcli: all describes on a first run, 0 when nothing changed, 1 for a changed, moved, or stale-row unit, a removed unit's output deleted, a failed describe retried, outputs byte-identical to `--full` and to a fresh full run, and the status line; 66 checks fail against master. `tests/wave2/bench-app-sweep.sh`, 2,000 units with 1 % changed: 20 describes incremental against 2,000 full. Field evidence: no field run; checked on synthetic and captured fixtures only. Not built: replacing describe with catalog queries (mxcli v0.25 `activities` with loop depth), NEEDS-FIELD-DATA, a real v0.25 catalog capture to check parity with the parser. — jopterhorst (#297) +- new(app-facts): **the app map's loop sweep is incremental, and `bin/status.sh` says how stale the map is** ([#297](https://github.com/MendixMau/mxcli-project-toolkit/issues/297)). Every run re-described every loop-containing microflow: on a 17,500-unit app that was 1,917 describes in 2 h 45 at 6 workers (about 31 s each, not the 1-2 s `app-analysis.md` promised), while about 80 units had changed between rounds. `manifest.json` now stores a `{UnitID: ContentsHash}` snapshot of the `.mpr` Unit table and an `mdl_index` recording the unit state each `mdl/` file was described from. A refresh describes only candidates whose hash, module or folder changed, or that have no output, keeps the rest and deletes outputs of removed or loop-free units. The sweep is full on the first run, with `--full`, after an mxcli version change, or when the Unit table cannot be read (MPR v1); `sections.loops.sweep` says which and why. UnitID-to-name mapping is `mx-metadata-note.py units`, the Studio Pro History note's own walk, refactored into a shared `Model` base for git revisions and the working tree. The hash is computed from the `.mxunit` bytes in the column's format, so a Unit-table row that lags its file cannot hide a change; disagreements are counted in the manifest. `bin/status.sh` prints `app map: N units changed since ` or `app map: fresh` when a map exists (`app-facts.sh --freshness`, no mxcli needed; the unit list goes through a temp file, never an env var, so it works past ~1,600 units), and nothing otherwise. A `--skip-loops` run keeps the last loop sweep's snapshot, so a stale loop map never reads fresh. `app-analysis.md` gives the measured cost by app size and names `--full` as the remedy after edits made outside Studio Pro. Fixture `tests/wave2/test-app-facts-incremental.sh`, 108 checks on both layouts with captured template units and a stub mxcli: all describes on a first run, 0 when nothing changed, 1 for a changed, moved, or stale-row unit, a removed unit's output deleted, a failed describe retried, a renamed module re-described under its new name, `--freshness` and status.sh on 2,101 units, a `--skip-loops` run not reading fresh, outputs byte-identical to `--full` and to a fresh full run, and the status line; 78 checks fail against master. `tests/wave2/bench-app-sweep.sh`, 2,000 units with 1 % changed: 20 describes incremental against 2,000 full, and `--freshness` reads fresh / 20 changed / fresh. Follow-up, not built: skip re-hashing unchanged `.mxunit` files by mtime + size. Field evidence: no field run; checked on synthetic and captured fixtures only. Not built: replacing describe with catalog queries (mxcli v0.25 `activities` with loop depth), NEEDS-FIELD-DATA, a real v0.25 catalog capture to check parity with the parser. — jopterhorst (#297) - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/project-bin/app-facts.sh b/project-bin/app-facts.sh index 94131e18..9a2a0576 100755 --- a/project-bin/app-facts.sh +++ b/project-bin/app-facts.sh @@ -236,25 +236,31 @@ DB="$MODEL_DIR/.mxcli/catalog.db" # --------------------------------------------------------------------------------------------- if [ "$FRESHNESS" -eq 1 ]; then [ -f "$FACTS_DIR/manifest.json" ] || exit 0 - NOW_TSV="$("$PY" "$UNITS_PY" units "$MODEL_DIR/$MPR_BASE" --hashes-only 2>/dev/null)" || NOW_TSV="__unreadable__" - NOW_TSV="$NOW_TSV" "$PY" - "$FACTS_DIR/manifest.json" <<'PY' -import json, os, sys + # The unit list goes through a temp file, never an env var or argv: one env string is capped at + # 128 KB on Linux, ~1,600 units' worth, and the line vanished on exactly the large apps (#297 review). + NOW_FILE="$(mktemp "${TMPDIR:-/tmp}/app-facts-units.XXXXXX")" || exit 0 + trap 'rm -f "$NOW_FILE"' EXIT + if "$PY" "$UNITS_PY" units "$MODEL_DIR/$MPR_BASE" --hashes-only > "$NOW_FILE" 2>/dev/null; then NOW_OK=1; else NOW_OK=0; fi + "$PY" - "$FACTS_DIR/manifest.json" "$NOW_FILE" "$NOW_OK" <<'PY' +import json, sys try: man = json.load(open(sys.argv[1])) except (OSError, ValueError): print("app map: manifest.json unreadable — rerun bin/app-facts.sh"); sys.exit(0) -when = (man.get("generated") or "?")[:16].replace("T", " ") -snap = (man.get("units") or {}).get("snapshot") -now = os.environ["NOW_TSV"] +units = man.get("units") or {} +# The snapshot's own date: a --skip-loops run carries the last loop sweep's snapshot forward. +when = (units.get("snapshot_taken") or man.get("generated") or "?")[:16].replace("T", " ") +snap = units.get("snapshot") if not snap: print("app map: built %s with no unit snapshot — rerun bin/app-facts.sh to track freshness" % when) -elif now == "__unreadable__": +elif sys.argv[3] != "1": print("app map: built %s; unit table unreadable, freshness unknown" % when) else: - cur = dict(l.split("\t", 1) for l in now.splitlines() if "\t" in l) + cur = dict(l.rstrip("\n").split("\t", 1) for l in open(sys.argv[2], encoding="utf-8") if "\t" in l) n = sum(1 for u, h in cur.items() if snap.get(u) != h) + sum(1 for u in snap if u not in cur) print("app map: fresh (built %s)" % when if n == 0 else "app map: %d units changed since %s" % (n, when)) PY + rm -f "$NOW_FILE"; trap - EXIT exit 0 fi mkdir -p "$FACTS_DIR/mdl" || exit 2 @@ -687,7 +693,7 @@ T1="$T1" T2="$T2" T3="$T3" TALL="$(( $(date +%s) - START_ALL ))" \ LOOP_STATUS="$LOOP_STATUS" LOOP_REASON="$LOOP_REASON" N_CAND="$N_CAND" N_DESC="$N_DESC" N_FAILED="$N_FAILED" N_RULES="$N_RULES" \ BUILD_MODE="$BUILD_MODE" N_MODULES="$N_MODULES" N_REFS="$N_REFS" N_ACTS="$N_ACTS" WORKERS="$WORKERS" MAX_LOOPS="$MAX_LOOPS" \ UNITS_TSV="$UNITS_TSV" UNITS_STATUS="$UNITS_STATUS" UNITS_REASON="$UNITS_REASON" PLAN="$PLAN" FULL="$FULL" \ -SWEEP_MODE="$SWEEP_MODE" SWEEP_REASON="$SWEEP_REASON" "$PY" - <<'PY' +SWEEP_MODE="$SWEEP_MODE" SWEEP_REASON="$SWEEP_REASON" SKIP_LOOPS="$SKIP_LOOPS" "$PY" - <<'PY' import json, os, datetime, hashlib F = os.environ["FACTS_DIR"] try: @@ -708,6 +714,18 @@ if snap and prev_snap: "changed": sum(1 for u, h in snap.items() if u in prev_snap and prev_snap[u] != h), "added": sum(1 for u in snap if u not in prev_snap), "removed": sum(1 for u in prev_snap if u not in snap)} +# The snapshot dates the map, and the loop section is its slowest part. A --skip-loops run leaves +# mdl/ as it was, so it keeps the snapshot (and its date) of the last loop sweep: otherwise +# --freshness would say "fresh" over stale loop outputs (#297 review). None yet: none stored. +gen = datetime.datetime.now().astimezone().isoformat(timespec="seconds") +units_note = "" +if os.environ["SKIP_LOOPS"] == "1": + snap_out = prev_snap + snap_taken = (prev.get("units") or {}).get("snapshot_taken") or prev.get("generated", "") if prev_snap else "" + units_note = ("; snapshot carried from the last loop sweep (--skip-loops)" if prev_snap + else "; no snapshot stored: --skip-loops and no earlier loop sweep") +else: + snap_out, snap_taken = snap, (gen if snap else "") # mdl_index: which unit state each mdl/ file was described from. Only files that exist now are # indexed, so a failed describe is retried next run. --skip-loops leaves mdl/ alone and carries the # previous index forward unchanged. @@ -728,15 +746,15 @@ loops = json.load(open(os.path.join(F, "loops.json"))) if os.path.exists(os.path mismatch = len(loops["_meta"]["parse_mismatch"]) if loops else 0 man = { "schema": "app-facts/1", - "generated": datetime.datetime.now().astimezone().isoformat(timespec="seconds"), + "generated": gen, "mpr": os.environ["MPR_BASE"], "mpr_sha256_16": sha(mp) if os.path.exists(mp) else "", "mxcli": os.environ["MX_VERSION"], "catalog": {"modules": int(os.environ["N_MODULES"]), "refs": int(os.environ["N_REFS"]), "activities": int(os.environ["N_ACTS"]), "build_mode": os.environ.get("BUILD_MODE", "")}, "timings_s": {"catalog_and_graph_report": int(os.environ["T1"]), "inventory_and_dependencies": int(os.environ["T2"]), "loops": int(os.environ["T3"]), "total": int(os.environ["TALL"])}, "options": {"workers": int(os.environ["WORKERS"]), "max_loops": int(os.environ["MAX_LOOPS"]), "full": os.environ["FULL"] == "1"}, - "units": {"status": os.environ["UNITS_STATUS"], "reason": os.environ["UNITS_REASON"], "count": len(snap), + "units": {"status": os.environ["UNITS_STATUS"], "reason": os.environ["UNITS_REASON"] + units_note, "count": len(snap_out), "source": "the .mpr Unit table; ContentsHash format, computed from each .mxunit's bytes", "changed_since_previous_map": changed, - "snapshot": snap}, + "snapshot_taken": snap_taken, "snapshot": snap_out}, "mdl_index": mdl_index, "mdl_index_mxcli": mdl_index_mxcli, "sections": { "inventory": {"status": "pass", "file": "inventory.json"}, diff --git a/skills/app-analysis.md b/skills/app-analysis.md index f0e6d890..8c54a331 100644 --- a/skills/app-analysis.md +++ b/skills/app-analysis.md @@ -35,6 +35,8 @@ carry them all across. describes in 2 h 45 at 6 workers, about 31 s each. Tell the user before a first run on a large app. A refresh re-describes only units whose hash, module or folder changed since their output was written (`manifest.json` → `sections.loops.sweep`); `--full` forces all. + Run `--full` after edits made outside Studio Pro (a scripted rename, say): a describe can + show another document's name, and only the described unit's own hash is compared. Exit 2 means the facts are not trustworthy; read the message, fix the cause, rerun. Do not write a dossier from a run that exited 2. 2. **Read `manifest.json` first.** Check `sections.*.status`. Note `parse_mismatch`, `failed` diff --git a/tests/wave2/bench-app-sweep.sh b/tests/wave2/bench-app-sweep.sh index 9b617194..4a588ae1 100755 --- a/tests/wave2/bench-app-sweep.sh +++ b/tests/wave2/bench-app-sweep.sh @@ -7,7 +7,8 @@ # Builds a synthetic model of N microflow units, every one loop-containing (the worst case: one # describe per unit), through fixtures/app-facts-incremental/ (a stub mxcli that counts describes), # then runs: a first map (full), a refresh with nothing changed, a refresh with P % of the units -# changed, and a --full refresh. The count of describes is the number that matters: on a real +# changed, and a --full refresh; and --freshness after the map, after the change and after the +# refresh (fresh / N changed / fresh) on a model past the 128 KB single-env-var size. The count of describes is the number that matters: on a real # model each one costs seconds to tens of seconds (~31 s each measured on a 17,500-unit app), and # the stub's wall time here says nothing about that. Exit 1 when the incremental refresh describes # anything other than exactly the changed units. @@ -39,13 +40,23 @@ run() { DESC=$(grep -c . "$DESCRIBE_LOG"); SECS=$(( $(date +%s) - t0 )) } +fresh() { ( cd "$P" && PROJECT_ROOT="$P" bash "$AF" --freshness --facts-dir "$P/facts" ) 2>&1; } + echo "bench-app-sweep: $N units, all loop-containing; $CHANGED changed ($PCT %)" printf '%-34s %10s %8s\n' "refresh" "describes" "wall s" build 1; run; printf '%-34s %10s %8s\n' "first map (full)" "$DESC" "$SECS"; D_FIRST=$DESC run; printf '%-34s %10s %8s\n' "refresh, nothing changed" "$DESC" "$SECS"; D_NONE=$DESC -build 2; run; printf '%-34s %10s %8s\n' "refresh, $PCT % changed (incremental)" "$DESC" "$SECS"; D_INC=$DESC +FR_FIRST="$(fresh)" +build 2; FR_STALE="$(fresh)" +run; printf '%-34s %10s %8s\n' "refresh, $PCT % changed (incremental)" "$DESC" "$SECS"; D_INC=$DESC +FR_AFTER="$(fresh)" run --full; printf '%-34s %10s %8s\n' "refresh, $PCT % changed (--full)" "$DESC" "$SECS"; D_FULL=$DESC echo "(wall time is the stub's; on a real model multiply describes by the per-describe cost)" +echo "freshness: after first map '$FR_FIRST' · after $PCT % changed '$FR_STALE' · after refresh '$FR_AFTER'" +case "$FR_FIRST|$FR_STALE|$FR_AFTER" in + "app map: fresh "*"|app map: $CHANGED units changed since "*"|app map: fresh "*) : ;; + *) echo "FAIL: --freshness did not report fresh / $CHANGED changed / fresh"; exit 1 ;; +esac if [ "$D_FIRST" -eq "$N" ] && [ "$D_NONE" -eq 0 ] && [ "$D_INC" -eq "$CHANGED" ] && [ "$D_FULL" -eq "$N" ]; then echo "OK: incremental describes $D_INC of $N ($PCT %), full describes $D_FULL" exit 0 diff --git a/tests/wave2/fixtures/app-facts-incremental/mkmodel.py b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py index 180e4bc8..4ec2471c 100755 --- a/tests/wave2/fixtures/app-facts-incremental/mkmodel.py +++ b/tests/wave2/fixtures/app-facts-incremental/mkmodel.py @@ -5,7 +5,7 @@ # python3 mkmodel.py [] # # spec.tsv, one microflow per line: -# All microflows live in MyFirstModule. changes the unit's bytes (so its ContentsHash) +# All microflows live in MyFirstModule ($MKMODEL_MODULE renames it: same UnitID, new Name). changes the unit's bytes (so its ContentsHash) # and the MDL the stub mxcli "describes" for it; moves the unit without changing them. # # WHAT IS CAPTURED AND WHAT IS NOT. With (tests/wave2/fixtures/ @@ -18,7 +18,7 @@ md, prep, spec = sys.argv[1:4] cap = sys.argv[4] if len(sys.argv) > 4 else None -MODNAME = "MyFirstModule" +MODNAME = os.environ.get("MKMODEL_MODULE", "MyFirstModule") # another name = a renamed module MOD = "16747dc6-b6b7-42ae-aabf-255dca2aeeaf" ROOT = bytes.fromhex("4d827743a25b514b800e658754fc369b") # the captured project root container CAPTURED_MF = {"MyFirstLogic": "4a8475b3-14cb-4100-8963-58f2fd3a0be8"} @@ -59,7 +59,7 @@ def bson(fields): flows.append((name, ver, None if folder == "-" else folder, loops == "1")) units = [] # (uid, container blob, containment, bytes) -mod_bytes = open(os.path.join(cap, MOD + ".mxunit"), "rb").read() if cap else \ +mod_bytes = open(os.path.join(cap, MOD + ".mxunit"), "rb").read() if cap and MODNAME == "MyFirstModule" else \ bson([("$Type", "Projects$ModuleImpl"), ("Name", MODNAME)]) units.append((MOD, ROOT, "Modules", mod_bytes)) for f in sorted({f for _, _, f, _ in flows if f}): diff --git a/tests/wave2/test-app-facts-incremental.sh b/tests/wave2/test-app-facts-incremental.sh index 111c0442..c0af312d 100755 --- a/tests/wave2/test-app-facts-incremental.sh +++ b/tests/wave2/test-app-facts-incremental.sh @@ -28,6 +28,9 @@ # T10 --freshness: "fresh", then "N units changed since "; silent with no map # T11 bin/status.sh shows the app map line (full screen and --brief); no line without a map # T12 the real captured units map to MyFirstModule.MyFirstLogic through mx-metadata-note.py units +# T13 --freshness and status.sh on 2,101 units (past the 128 KB single-env-var cap) +# T14 a renamed module: old outputs deleted, every candidate re-described under the new name +# T15 a --skip-loops run keeps the last loop sweep's snapshot, so freshness is not "fresh" set -uo pipefail TK="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" @@ -213,6 +216,27 @@ NEW_One 1 - 1" mv "$W/man.bak" "$FACTS/manifest.json" fi + # --- T15 --skip-loops keeps the last loop sweep's snapshot ----------------------------------- + # The model has 2 changed units since the map; a --skip-loops run leaves mdl/ stale, so the + # freshness line must not turn "fresh" (#297 review). + run --skip-loops + case "$(fresh)" in "app map: 2 units changed since "*) ok "[$layout] T15 after --skip-loops the line still counts 2 changed units" ;; + *) bad "[$layout] T15 --skip-loops made a stale loop map read fresh" "$(fresh)" ;; esac + run + case "$(fresh)" in "app map: fresh (built "*) ok "[$layout] T15 a loop sweep then makes it fresh" ;; + *) bad "[$layout] T15 freshness after the loop sweep" "$(fresh)" ;; esac + + # --- T14 module renamed: same UnitIDs and microflow bytes, new qualified names --------------- + SPEC_REN="$(printf '%s\n' "$SPEC_NOW" | sed 's/^SUB_A 1/SUB_A 3/') +NEW_One 1 - 1" + MKMODEL_MODULE=Renamed model "$SPEC_REN" + run + check "[$layout] T14 renamed module: every loop microflow re-described (5)" "$NDESC" "5" + check "[$layout] T14 no output left under the old module name" "$(ls "$FACTS/mdl" | grep -c "^$M\.")" "0" + check "[$layout] T14 5 outputs under the new name" "$(ls "$FACTS/mdl" | grep -c '^Renamed\.')" "5" + check "[$layout] T14 loops.json lists the new names only" \ + "$("$PY" -c 'import json,sys;print(sorted({k.split(".")[0] for k in json.load(open(sys.argv[1]))["microflows"]}))' "$FACTS/loops.json")" "['Renamed']" + # --- T9 unreadable Unit table ---------------------------------------------------------------- P9="$W/$layout-v1"; mkdir -p "$P9"; if [ "$layout" = two ]; then MD9="$P9/app"; else MD9="$P9"; fi MD_SAVE="$MD"; P_SAVE="$P"; FACTS_SAVE="$FACTS" @@ -237,6 +261,23 @@ check "T12 hash is the Unit table's ContentsHash = base64(sha256(.mxunit))" "$(p check "T12 the module unit itself has no qualified name" \ "$("$PY" "$SRC/project-bin/mx-metadata-note.py" units "$MD/App.mpr" | grep '^16747dc6' | cut -f3,4)" "Projects\$ModuleImpl " +# --- T13 --freshness on a model of more than 2,000 units --------------------------------------- +# The unit list once went to Python through one env var, capped at 128 KB on Linux (~1,600 units): +# "Argument list too long", and status.sh, which hides stderr, silently dropped the line. +echo "== T13 --freshness on 2,100 units" +P="$W/big"; MD="$P"; PREP_DB="$W/big.db"; DESCRIBE_LOG="$W/big.log"; FACTS="$P/analysis/app-facts"; mkdir -p "$P" +BIG="$(awk 'BEGIN { for (i = 1; i <= 2100; i++) printf "ACT_%05d\t1\t-\t0\n", i }')" +model "$BIG"; run +check "T13 map of 2,100 units built" "$(jq_ "$FACTS/manifest.json" units/count)" "2101" +F13="$( (cd "$P" && PROJECT_ROOT="$P" bash "$AF" --freshness --facts-dir "$FACTS") 2>&1)" +case "$F13" in "app map: fresh (built "*) ok "T13 2,101 units: freshness says fresh" ;; *) bad "T13 freshness on 2,101 units" "$F13" ;; esac +model "$(printf '%s\n' "$BIG" | sed 's/^ACT_00007 1/ACT_00007 2/')" +F13="$( (cd "$P" && PROJECT_ROOT="$P" bash "$AF" --freshness --facts-dir "$FACTS") 2>&1)" +case "$F13" in "app map: 1 units changed since "*) ok "T13 2,101 units, one changed: counted" ;; *) bad "T13 changed count on 2,101 units" "$F13" ;; esac +printf '# PROJECT\n\n## Current stage\n**Stage 1 in progress.**\n' > "$P/PROJECT.md" +bash "$STATUS" "$P" --brief 2>/dev/null | grep -q "app map: 1 units changed since " && ok "T13 status.sh --brief carries the line on 2,101 units" \ + || bad "T13 status.sh lost the app map line on a large model" + echo "" echo "PASS=$PASS FAIL=$FAIL" [ "$FAIL" -eq 0 ]