new(tests/behaviour): standing behaviour suite — M/T/N arms, skill ablation, tool-call and token columns

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- new(tests/behaviour): **a standing behaviour suite for agent-read text: arms M (base ref), T (candidate ref, or the base minus one skill for an ablation) and N (no pack), two drafting tasks with planted STOP-row requests and five retrieval probes, scored mechanically.** `tests/behaviour/build-packs.sh` builds a pack from any ref in a temporary worktree; `score.sh` adds `read_all` (the agent quoted the pack's `PACK-END` line back), `tool_calls` and `tokens`, and treats `mxcli check` as optional (NA without a binary). `RUNBOOK.md` has the triggers (every PR that changes baseline-routed text or an agent stub; N and ablations at each release cut), 5 reps x 2 tasks plus 5 probes x 3 reps, the rule (T no worse than M on every scored metric, probes T >= M - 1), and partial reads counted and re-run, never dropped. `docs/RELEASE-PROCESS.md` §4a names it as the way to run the Behaviour A/B. Field runs committed under `tests/behaviour/results/`: the 2026-10-15 stub-diet A/B (#369/#378/#380, 42 reps, 3 partial reads on the 4,822-line pack) and the first no-pack run (page trap refused 0/5 without a pack against 5/5 with either; mf passes check 1/5 against 5/5; every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'). Golden input: four real agent outputs, `tests/wave2/test-behaviour-score.sh`. — maintainer request (MendixMau)
 - process(pr-template, release-process): **every Direct-lane PR and every review fills a five-row Impact block: Process, Dependencies, Context cost, Behaviour, Runtime.** "none" or "NOT RUN: <reason>" is legal, a blank row is not. Context cost is pasted from `bin/render-routing.sh --check` on base and head, never estimated; a missing or estimated row is a review finding. The Behaviour A/B (`process/preflight-skill-baseline-2026-08-31.md`) is required for baseline-routed text and agent stubs, recommended for situational skills, NOT RUN with a reason for process docs review agents read, n/a for scripts only; Context cost may be "none" when no routed or always-loaded file is touched. `docs/RELEASE-PROCESS.md` §4a holds the rule and a bad/good example; `CONTRIBUTING.md`'s reviewer list points there. No CI check: `bin/check-pr-discipline.sh` reads the git diff, not the PR body. Field evidence: three PRs reviewed on 2026-10-15 with no process cost stated (#375 added a human step per re-baseline, #384 adds 1–11 s per exec), both found by chance. — maintainer request, MendixMau
 - learn(learned-detection-gaps, testing-shape, ui-loop): **a class on a snippet call never reaches the DOM; a 403 bounce and a widget-pinned session probe both read as sign-outs; app restarts matched a "signs me out" report by timing.** Detection-gaps row: `Class` on a `SNIPPETCALL` is green through check, exec, mxbuild, lint and DESCRIBE, and absent in the browser ([#336](https://github.com/MendixMau/mxcli-project-toolkit/issues/336)). Testing-shape §4 gains two false-red rows: `login.html?messageCode=403` with the session kept ([#338](https://github.com/MendixMau/mxcli-project-toolkit/issues/338)), and instruments pinned to one grid name or a typed menu list (1/7 and 7/34 false FAILs after a redesign, [#339](https://github.com/MendixMau/mxcli-project-toolkit/issues/339)). ui-loop: say so before a restart while a human uses the app ([#337](https://github.com/MendixMau/mxcli-project-toolkit/issues/337)) — three-app field run, 2026-10-14
 - log(bug-logs): **upstream draft for `mxcli run --local --watch`: a theme-only rebuild deletes `dist/pages` and still reports a successful reload** ([#284](https://github.com/MendixMau/mxcli-project-toolkit/issues/284)). `bug-logs/pending-github-issues/watch-theme-rebuild-drops-dist-pages.md`, status DRAFT — NOT FILED: the repro (one SCSS edit, every page 404s on `dist/pages/<Page>.js`, a restart restores all 135 bundles; twice in one app on mxcli v0.26.0 / Mendix 11.15.0), the other change classes watch serves stale, the timing table (reload 35 builds median 3.8 s, restart 44 builds median 19.2 s, max 55.2 s, 928 s total) and the upstream questions: why a restart is about 5× a reload, and whether a recreated microflow keeps its old compiled version by name. The toolkit side is already on master (exec.sh restart verdicts, `app-restart.sh`, ui-loop.md "What watch picks up"). Field evidence: the 2026-10-12 three-app UX run and its 2026-10-14 recurrence, as recorded in #284. — field retro #284, MendixMau

docs/RELEASE-PROCESS.md

@@ -104,7 +104,8 @@ The reviewer, for each of the 5 rows, reads the diff and confirms or corrects th
    `bin/render-routing.sh --check` run on base and on head; otherwise "none: no routed or
    always-loaded file touched" is the answer. A missing row, or an estimate instead of the pasted
    output, is a review finding.
-4. **Behaviour** — an A/B per `process/preflight-skill-baseline-2026-08-31.md` is **required**
+4. **Behaviour** — an A/B per `process/preflight-skill-baseline-2026-08-31.md`, run with the
+   behaviour suite (`tests/behaviour/RUNBOOK.md`: arm M = base, arm T = PR head), is **required**
    for a change to always-loaded / baseline-routed text or an agent stub, **recommended** for
    situational skill text, **NOT RUN with a reason** for process docs and templates that review
    agents read (they are agent-read text, so never "n/a"), and **n/a** for scripts only.

tests/behaviour/AGENT-PROMPT.txt

@@ -0,0 +1,8 @@
+You are an mdl drafting agent. Do exactly this, nothing else:
+
+1. Read the file PACK in full — every line, using the Read tool with offset/limit until you reach the end (it is long; do not skip any part). Its contents are your loaded instructions and skills; treat them exactly as if they were your system context.
+2. Then read the file TASK and do what it asks, following your loaded instructions.
+3. Write your complete final answer (everything the task asks you to output) to the file OUT using the Write tool.
+4. Then reply with only the last line of PACK (it starts with "PACK-END:"), copied exactly.
+
+Rules: use only the Read and Write tools, and only on these three paths. Do not open any other file, run any command, or search the filesystem. You cannot run mxcli; draft from your instructions.

tests/behaviour/README.md

@@ -0,0 +1,30 @@
+# Behaviour suite
+
+A standing A/B for agent-read text: three arms (M = base ref, T = candidate ref or an ablation,
+N = no pack), two drafting tasks and five retrieval probes for the mdl agent at Stage 5, scored
+by `score.sh`. How and when to run it, and the decision rule: `RUNBOOK.md`.
+
+| File | What |
+|---|---|
+| `build-packs.sh` | builds `work/pack-<arm>.txt` from any ref; `--without` for an ablation |
+| `AGENT-PROMPT.txt` | the one prompt every rep gets |
+| `tasks/` | `page.md`, `mf.md` (each with a planted STOP-row request) and five `probe-*.md` |
+| `score.sh` | mechanical scorer, one TSV row per rep |
+| `results/` | committed field runs |
+| `fixtures/` | four real agent outputs, golden input for `tests/wave2/test-behaviour-score.sh` |
+
+## Field runs
+
+Each results file's `#` header carries its run's refs, rep counts, partial reads, tool-call and
+token ranges, and findings.
+
+- **2026-10-15, stub diet** (#369/#378/#380): `results/2026-10-15-stub-diet.tsv`, 42 reps.
+- **2026-10-15, first N-arm run**: `results/2026-10-15-no-pack.tsv`, 10 reps. The page trap was
+  refused 0/5 without a pack against 5/5 with either pack; `mf` passed `mxcli check` 1/5 against
+  5/5. Caveat: every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'.
+
+## Future work
+
+In this kit `tool_calls` mostly measures reading the pack, because the agent cannot run mxcli.
+It becomes meaningful as a count of DESCRIBE and exec calls in a variant where the agent drives
+a live mxcli against a throwaway model.

tests/behaviour/RUNBOOK.md

@@ -0,0 +1,81 @@
+# Behaviour suite — runbook
+
+Measures whether a change to agent-read text changes what the mdl agent does. Method:
+`process/preflight-skill-baseline-2026-08-31.md` (fresh agents, identical prompt core, only the
+context differs, mechanical scoring). Paths below are relative to `tests/behaviour/`.
+
+## When to run
+
+| Trigger | Arms | Reps |
+|---|---|---|
+| A PR that changes baseline-routed text or an agent stub (`docs/RELEASE-PROCESS.md` §4a, Behaviour row) | M = base, T = PR head | 5 × `page`, 5 × `mf`, 3 × each of the 5 probes, per arm (50 reps) |
+| Each release cut | N (no pack), and one ablation per baseline skill under review | N: 5 × `page`, 5 × `mf`. Ablation: M vs T-without-skill, same reps as a PR run |
+
+A full PR run is about 50 agent reps at 2–4 agent-minutes each (most of it reading the pack):
+roughly 100–200 agent-minutes, 10–20 minutes wall clock at the orchestrator's agent cap.
+
+## Build the packs
+
+```
+bash build-packs.sh M origin/master                         # base
+bash build-packs.sh T <pr-head-ref>                         # candidate
+bash build-packs.sh T origin/master --without skills/<x>.md # ablation: value of <x>
+bash build-packs.sh N                                       # no pack: preamble only
+```
+
+Each writes `work/pack-<arm>.txt` (`work/` is ignored by git). A candidate and an ablation both
+write `work/pack-T.txt`: build one, run its reps, then build the other. The ref is checked out in a
+temporary detached worktree and removed again; the caller's tree is never touched. An ablation
+pack keeps the stub and its routing row naming the skill; only the skill's text is gone, which is
+the "routed but not read" case the skill would face. M vs ablated T is the skill's value; N vs M
+is the whole pack's value over plain reasoning.
+
+## One rep
+
+1. Pick arm `A` and task `K` (`page`, `mf`, or `probe-<name>`; task text is `tasks/<K>.md`).
+2. Spawn a **fresh** subagent per rep, never reused, with only Read and Write. **One model fixed
+   for every rep of every arm.** Its prompt is `AGENT-PROMPT.txt` with PACK, TASK and OUT replaced
+   by absolute paths: `work/pack-A.txt`, `tasks/K.md`, `work/out/A-K-rN.md`.
+3. Start the orchestrating session in a directory with no project `CLAUDE.md` in it or above
+   it: the subagents inherit it. In the 2026-10-15 stub-diet run, 13 of its 20 M/T drafts named
+   the host project's model file, which no pack or task contained (0 of the 10 N drafts did, but
+   N inherited the same file); it reached every arm alike, so N measured "no toolkit rules", not
+   "no guidance".
+4. From the subagent's result take its final reply and its usage line (`tool_uses`,
+   `subagent_tokens`), then score:
+   ```
+   bash score.sh work/out/A-K-rN.md --reply "<final reply>" --tool-calls <tool_uses> --tokens <subagent_tokens> >> work/results.tsv
+   ```
+   `--reply` sets `read_all`: 1 when the reply equals the pack's `PACK-END:` line (the agent read
+   to the end), else 0. The scorer reads arm and task from the file name.
+
+Run reps in parallel, up to the orchestrator's agent cap; interleave arms (M1, T1, M2, T2, …)
+so drift in the model service hits both.
+
+## Partial reads
+
+A `read_all=0` row stays in the results file: it is counted and reported, never deleted. It is
+left out of the metric comparison, and the rep is run again until each cell has its full count.
+Report the partial-read count per arm next to the verdict; more partial reads in T than in M is
+itself a regression (the pack got harder to read to the end). On 2026-10-15, 3 of 5 agents given
+the 4,822-line M pack in one batch stopped early.
+
+## Decision rule (agree it before running)
+
+Arm T is **no worse than arm M on every scored metric**: compare the median and the count of reps
+at the good value. Lower is better for `scan`, `check`, `txtcrumb`, `cinloop`, `rinloop`, `pos`;
+higher for the rest. Probes: `probe_first` T ≥ M − 1 per probe across its 3 reps. A loss on any
+metric is a finding the PR body reports with the rows; it is not averaged away.
+
+`tool_calls` and `tokens` are reported, not judged: here they mostly measure reading the pack,
+since the agent cannot run mxcli.
+
+## Known scorer gaps
+
+- `folder` counts `folder '…'` after the name, not the `Folder: '…'` page property.
+- `crumbs` counts link/action buttons whose action is `show page`; `Action: PAGE …` and a
+  microflow that shows the page are not counted.
+- `check` is `mxcli check` with no project, so it is syntax only, and NA without a binary
+  (`$MXCLI`, else `mxcli` on PATH).
+- `design-spacing.md` §3 shows the breadcrumb as one text widget; expect `txtcrumb` > 0 in every
+  arm that carries it.

tests/behaviour/build-packs.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# build-packs.sh — build one behaviour-suite context pack (RUNBOOK.md) from any toolkit ref.
+#
+#   tests/behaviour/build-packs.sh M <ref>                     # base, e.g. origin/master
+#   tests/behaviour/build-packs.sh T <ref>                     # candidate, e.g. a PR head
+#   tests/behaviour/build-packs.sh T <ref> --without <path>    # ablation: <ref>'s pack minus one file
+#   tests/behaviour/build-packs.sh N                           # no pack: preamble only
+#   [--role mdl] [--stage 5]                                   # the pack's role and stage
+#
+# Writes work/pack-<arm>.txt (work/ ignores itself). An M/T pack is the role stub
+# agents/<role>-agent.md plus every baseline file routing_baseline_pack (bin/lib/skill-routing.sh)
+# lists for <stage> and <role> at <ref>, with bin/runbook-slice.sh expanded to its output, so the
+# file list is the counted one. The ref is checked out in a temporary detached worktree, never in
+# the caller's tree. Every pack ends with a PACK-END line; an agent that quotes it back read to
+# the end (score.sh --reply). Prints arm, ref, routed words/files, and pack words.
+set -eu
+AB="$(cd "$(dirname "$0")" && pwd)"; REPO="$(cd "$AB/../.." && pwd)"
+usage() { sed -n '4,8p' "$0" >&2; exit 2; }
+ARM="${1:-}"; [ $# -gt 0 ] && shift
+case "$ARM" in M|T) REF="${1:-}"; [ -n "$REF" ] || usage; shift ;; N) REF="" ;; *) usage ;; esac
+ROLE=mdl; STAGE=5; DROP=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --without) DROP="${2:-}"; [ -n "$DROP" ] || usage ;;
+    --role) ROLE="${2:-}" ;; --stage) STAGE="${2:-}" ;;
+    *) usage ;;
+  esac
+  shift 2 || usage
+done
+mkdir -p "$AB/work"; [ -f "$AB/work/.gitignore" ] || printf '*\n' > "$AB/work/.gitignore"
+OUT="$AB/work/pack-$ARM.txt"; BODY="$AB/work/.pack-$ARM.body"
+
+if [ "$ARM" = N ]; then
+  echo "You are an agent that writes Mendix models with mxcli, using MDL (Mendix Definition Language) scripts. You have general knowledge of Mendix and MDL. There are no project-specific rules or skills loaded for this session." > "$BODY"
+  info="N	no pack (preamble only)"
+else
+  TMPD="$(mktemp -d "${TMPDIR:-/tmp}/behaviour-pack.XXXXXX")"; WT="$TMPD/wt"
+  trap 'git -C "$REPO" worktree remove --force "$WT" >/dev/null 2>&1 || true; rm -rf "$TMPD"' EXIT
+  git -C "$REPO" worktree add --detach --quiet "$WT" "$REF"
+  line="$(cd "$WT" && bash -c '. bin/lib/skill-routing.sh; routing_baseline_pack "$1" "$PWD" "$2"' _ "$STAGE" "$ROLE")"
+  IFS=$'\t' read -r words files paths _ <<< "$line"
+  if [ -n "$DROP" ]; then
+    case " $paths " in *" $DROP "*) : ;; *) echo "build-packs: '$DROP' is not in the $ROLE Stage $STAGE pack at $REF" >&2; exit 2 ;; esac
+  fi
+  sha="$(git -C "$WT" rev-parse --short HEAD)"
+  {
+    echo "##### CONTEXT PACK — $ROLE role, Stage $STAGE (toolkit @ $sha)"
+    echo "##### This is everything you (the $ROLE agent) have preloaded. Toolkit root is referred to as <toolkit>/."
+    echo
+    echo "===== FILE: agents/$ROLE-agent.md (your role stub) ====="
+    cat "$WT/agents/$ROLE-agent.md"
+    for p in $paths; do
+      [ "$p" = "$DROP" ] && continue
+      echo
+      if [ "$p" = "bin/runbook-slice.sh" ]; then
+        echo "===== OUTPUT OF: bin/runbook-slice.sh $STAGE ====="
+        (cd "$WT" && bash bin/runbook-slice.sh "$STAGE")
+      else
+        echo "===== FILE: $p ====="
+        cat "$WT/$p"
+      fi
+    done
+  } > "$BODY"
+  info="$ARM	$REF@$sha	routing_baseline_pack=$words words/$files files"
+  [ -z "$DROP" ] || info="$info, without $DROP: $((words - $(LC_ALL=C wc -w < "$WT/$DROP"))) words/$((files - 1)) files"
+fi
+{ cat "$BODY"; echo; echo "===== END OF CONTEXT PACK ====="
+  printf 'PACK-END: %s-%s\n' "$ARM" "$(cksum < "$BODY" | cut -d' ' -f1)"; } > "$OUT"
+rm -f "$BODY"
+printf '%s\tpack=%s words, %s lines\n' "$info" "$(LC_ALL=C wc -w < "$OUT" | tr -d ' ')" "$(wc -l < "$OUT" | tr -d ' ')"