diff --git a/CHANGELOG.md b/CHANGELOG.md index 08db3eda..994f1202 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- new(tests/behaviour): **a standing behaviour suite for agent-read text: arms M (base ref), T (candidate ref, or the base minus one skill for an ablation) and N (no pack), two drafting tasks with planted STOP-row requests and five retrieval probes, scored mechanically.** `tests/behaviour/build-packs.sh` builds a pack from any ref in a temporary worktree; `score.sh` adds `read_all` (the agent quoted the pack's `PACK-END` line back), `tool_calls` and `tokens`, and treats `mxcli check` as optional (NA without a binary). `RUNBOOK.md` has the triggers (every PR that changes baseline-routed text or an agent stub; N and ablations at each release cut), 5 reps x 2 tasks plus 5 probes x 3 reps, the rule (T no worse than M on every scored metric, probes T >= M - 1), and partial reads counted and re-run, never dropped. `docs/RELEASE-PROCESS.md` §4a names it as the way to run the Behaviour A/B. Field runs committed under `tests/behaviour/results/`: the 2026-10-15 stub-diet A/B (#369/#378/#380, 42 reps, 3 partial reads on the 4,822-line pack) and the first no-pack run (page trap refused 0/5 without a pack against 5/5 with either; mf passes check 1/5 against 5/5; every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'). Golden input: four real agent outputs, `tests/wave2/test-behaviour-score.sh`. — maintainer request (MendixMau) - process(pr-template, release-process): **every Direct-lane PR and every review fills a five-row Impact block: Process, Dependencies, Context cost, Behaviour, Runtime.** "none" or "NOT RUN: " is legal, a blank row is not. Context cost is pasted from `bin/render-routing.sh --check` on base and head, never estimated; a missing or estimated row is a review finding. The Behaviour A/B (`process/preflight-skill-baseline-2026-08-31.md`) is required for baseline-routed text and agent stubs, recommended for situational skills, NOT RUN with a reason for process docs review agents read, n/a for scripts only; Context cost may be "none" when no routed or always-loaded file is touched. `docs/RELEASE-PROCESS.md` §4a holds the rule and a bad/good example; `CONTRIBUTING.md`'s reviewer list points there. No CI check: `bin/check-pr-discipline.sh` reads the git diff, not the PR body. Field evidence: three PRs reviewed on 2026-10-15 with no process cost stated (#375 added a human step per re-baseline, #384 adds 1–11 s per exec), both found by chance. — maintainer request, MendixMau - learn(learned-detection-gaps, testing-shape, ui-loop): **a class on a snippet call never reaches the DOM; a 403 bounce and a widget-pinned session probe both read as sign-outs; app restarts matched a "signs me out" report by timing.** Detection-gaps row: `Class` on a `SNIPPETCALL` is green through check, exec, mxbuild, lint and DESCRIBE, and absent in the browser ([#336](https://github.com/MendixMau/mxcli-project-toolkit/issues/336)). Testing-shape §4 gains two false-red rows: `login.html?messageCode=403` with the session kept ([#338](https://github.com/MendixMau/mxcli-project-toolkit/issues/338)), and instruments pinned to one grid name or a typed menu list (1/7 and 7/34 false FAILs after a redesign, [#339](https://github.com/MendixMau/mxcli-project-toolkit/issues/339)). ui-loop: say so before a restart while a human uses the app ([#337](https://github.com/MendixMau/mxcli-project-toolkit/issues/337)) — three-app field run, 2026-10-14 - log(bug-logs): **upstream draft for `mxcli run --local --watch`: a theme-only rebuild deletes `dist/pages` and still reports a successful reload** ([#284](https://github.com/MendixMau/mxcli-project-toolkit/issues/284)). `bug-logs/pending-github-issues/watch-theme-rebuild-drops-dist-pages.md`, status DRAFT — NOT FILED: the repro (one SCSS edit, every page 404s on `dist/pages/.js`, a restart restores all 135 bundles; twice in one app on mxcli v0.26.0 / Mendix 11.15.0), the other change classes watch serves stale, the timing table (reload 35 builds median 3.8 s, restart 44 builds median 19.2 s, max 55.2 s, 928 s total) and the upstream questions: why a restart is about 5× a reload, and whether a recreated microflow keeps its old compiled version by name. The toolkit side is already on master (exec.sh restart verdicts, `app-restart.sh`, ui-loop.md "What watch picks up"). Field evidence: the 2026-10-12 three-app UX run and its 2026-10-14 recurrence, as recorded in #284. — field retro #284, MendixMau diff --git a/docs/RELEASE-PROCESS.md b/docs/RELEASE-PROCESS.md index 28398277..e3d592c4 100644 --- a/docs/RELEASE-PROCESS.md +++ b/docs/RELEASE-PROCESS.md @@ -104,7 +104,8 @@ The reviewer, for each of the 5 rows, reads the diff and confirms or corrects th `bin/render-routing.sh --check` run on base and on head; otherwise "none: no routed or always-loaded file touched" is the answer. A missing row, or an estimate instead of the pasted output, is a review finding. -4. **Behaviour** — an A/B per `process/preflight-skill-baseline-2026-08-31.md` is **required** +4. **Behaviour** — an A/B per `process/preflight-skill-baseline-2026-08-31.md`, run with the + behaviour suite (`tests/behaviour/RUNBOOK.md`: arm M = base, arm T = PR head), is **required** for a change to always-loaded / baseline-routed text or an agent stub, **recommended** for situational skill text, **NOT RUN with a reason** for process docs and templates that review agents read (they are agent-read text, so never "n/a"), and **n/a** for scripts only. diff --git a/tests/behaviour/AGENT-PROMPT.txt b/tests/behaviour/AGENT-PROMPT.txt new file mode 100644 index 00000000..c97a1804 --- /dev/null +++ b/tests/behaviour/AGENT-PROMPT.txt @@ -0,0 +1,8 @@ +You are an mdl drafting agent. Do exactly this, nothing else: + +1. Read the file PACK in full — every line, using the Read tool with offset/limit until you reach the end (it is long; do not skip any part). Its contents are your loaded instructions and skills; treat them exactly as if they were your system context. +2. Then read the file TASK and do what it asks, following your loaded instructions. +3. Write your complete final answer (everything the task asks you to output) to the file OUT using the Write tool. +4. Then reply with only the last line of PACK (it starts with "PACK-END:"), copied exactly. + +Rules: use only the Read and Write tools, and only on these three paths. Do not open any other file, run any command, or search the filesystem. You cannot run mxcli; draft from your instructions. diff --git a/tests/behaviour/README.md b/tests/behaviour/README.md new file mode 100644 index 00000000..2dcae77d --- /dev/null +++ b/tests/behaviour/README.md @@ -0,0 +1,30 @@ +# Behaviour suite + +A standing A/B for agent-read text: three arms (M = base ref, T = candidate ref or an ablation, +N = no pack), two drafting tasks and five retrieval probes for the mdl agent at Stage 5, scored +by `score.sh`. How and when to run it, and the decision rule: `RUNBOOK.md`. + +| File | What | +|---|---| +| `build-packs.sh` | builds `work/pack-.txt` from any ref; `--without` for an ablation | +| `AGENT-PROMPT.txt` | the one prompt every rep gets | +| `tasks/` | `page.md`, `mf.md` (each with a planted STOP-row request) and five `probe-*.md` | +| `score.sh` | mechanical scorer, one TSV row per rep | +| `results/` | committed field runs | +| `fixtures/` | four real agent outputs, golden input for `tests/wave2/test-behaviour-score.sh` | + +## Field runs + +Each results file's `#` header carries its run's refs, rep counts, partial reads, tool-call and +token ranges, and findings. + +- **2026-10-15, stub diet** (#369/#378/#380): `results/2026-10-15-stub-diet.tsv`, 42 reps. +- **2026-10-15, first N-arm run**: `results/2026-10-15-no-pack.tsv`, 10 reps. The page trap was + refused 0/5 without a pack against 5/5 with either pack; `mf` passed `mxcli check` 1/5 against + 5/5. Caveat: every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'. + +## Future work + +In this kit `tool_calls` mostly measures reading the pack, because the agent cannot run mxcli. +It becomes meaningful as a count of DESCRIBE and exec calls in a variant where the agent drives +a live mxcli against a throwaway model. diff --git a/tests/behaviour/RUNBOOK.md b/tests/behaviour/RUNBOOK.md new file mode 100644 index 00000000..9690830e --- /dev/null +++ b/tests/behaviour/RUNBOOK.md @@ -0,0 +1,81 @@ +# Behaviour suite — runbook + +Measures whether a change to agent-read text changes what the mdl agent does. Method: +`process/preflight-skill-baseline-2026-08-31.md` (fresh agents, identical prompt core, only the +context differs, mechanical scoring). Paths below are relative to `tests/behaviour/`. + +## When to run + +| Trigger | Arms | Reps | +|---|---|---| +| A PR that changes baseline-routed text or an agent stub (`docs/RELEASE-PROCESS.md` §4a, Behaviour row) | M = base, T = PR head | 5 × `page`, 5 × `mf`, 3 × each of the 5 probes, per arm (50 reps) | +| Each release cut | N (no pack), and one ablation per baseline skill under review | N: 5 × `page`, 5 × `mf`. Ablation: M vs T-without-skill, same reps as a PR run | + +A full PR run is about 50 agent reps at 2–4 agent-minutes each (most of it reading the pack): +roughly 100–200 agent-minutes, 10–20 minutes wall clock at the orchestrator's agent cap. + +## Build the packs + +``` +bash build-packs.sh M origin/master # base +bash build-packs.sh T # candidate +bash build-packs.sh T origin/master --without skills/.md # ablation: value of +bash build-packs.sh N # no pack: preamble only +``` + +Each writes `work/pack-.txt` (`work/` is ignored by git). A candidate and an ablation both +write `work/pack-T.txt`: build one, run its reps, then build the other. The ref is checked out in a +temporary detached worktree and removed again; the caller's tree is never touched. An ablation +pack keeps the stub and its routing row naming the skill; only the skill's text is gone, which is +the "routed but not read" case the skill would face. M vs ablated T is the skill's value; N vs M +is the whole pack's value over plain reasoning. + +## One rep + +1. Pick arm `A` and task `K` (`page`, `mf`, or `probe-`; task text is `tasks/.md`). +2. Spawn a **fresh** subagent per rep, never reused, with only Read and Write. **One model fixed + for every rep of every arm.** Its prompt is `AGENT-PROMPT.txt` with PACK, TASK and OUT replaced + by absolute paths: `work/pack-A.txt`, `tasks/K.md`, `work/out/A-K-rN.md`. +3. Start the orchestrating session in a directory with no project `CLAUDE.md` in it or above + it: the subagents inherit it. In the 2026-10-15 stub-diet run, 13 of its 20 M/T drafts named + the host project's model file, which no pack or task contained (0 of the 10 N drafts did, but + N inherited the same file); it reached every arm alike, so N measured "no toolkit rules", not + "no guidance". +4. From the subagent's result take its final reply and its usage line (`tool_uses`, + `subagent_tokens`), then score: + ``` + bash score.sh work/out/A-K-rN.md --reply "" --tool-calls --tokens >> work/results.tsv + ``` + `--reply` sets `read_all`: 1 when the reply equals the pack's `PACK-END:` line (the agent read + to the end), else 0. The scorer reads arm and task from the file name. + +Run reps in parallel, up to the orchestrator's agent cap; interleave arms (M1, T1, M2, T2, …) +so drift in the model service hits both. + +## Partial reads + +A `read_all=0` row stays in the results file: it is counted and reported, never deleted. It is +left out of the metric comparison, and the rep is run again until each cell has its full count. +Report the partial-read count per arm next to the verdict; more partial reads in T than in M is +itself a regression (the pack got harder to read to the end). On 2026-10-15, 3 of 5 agents given +the 4,822-line M pack in one batch stopped early. + +## Decision rule (agree it before running) + +Arm T is **no worse than arm M on every scored metric**: compare the median and the count of reps +at the good value. Lower is better for `scan`, `check`, `txtcrumb`, `cinloop`, `rinloop`, `pos`; +higher for the rest. Probes: `probe_first` T ≥ M − 1 per probe across its 3 reps. A loss on any +metric is a finding the PR body reports with the rows; it is not averaged away. + +`tool_calls` and `tokens` are reported, not judged: here they mostly measure reading the pack, +since the agent cannot run mxcli. + +## Known scorer gaps + +- `folder` counts `folder '…'` after the name, not the `Folder: '…'` page property. +- `crumbs` counts link/action buttons whose action is `show page`; `Action: PAGE …` and a + microflow that shows the page are not counted. +- `check` is `mxcli check` with no project, so it is syntax only, and NA without a binary + (`$MXCLI`, else `mxcli` on PATH). +- `design-spacing.md` §3 shows the breadcrumb as one text widget; expect `txtcrumb` > 0 in every + arm that carries it. diff --git a/tests/behaviour/build-packs.sh b/tests/behaviour/build-packs.sh new file mode 100755 index 00000000..dd6ad08a --- /dev/null +++ b/tests/behaviour/build-packs.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# build-packs.sh — build one behaviour-suite context pack (RUNBOOK.md) from any toolkit ref. +# +# tests/behaviour/build-packs.sh M # base, e.g. origin/master +# tests/behaviour/build-packs.sh T # candidate, e.g. a PR head +# tests/behaviour/build-packs.sh T --without # ablation: 's pack minus one file +# tests/behaviour/build-packs.sh N # no pack: preamble only +# [--role mdl] [--stage 5] # the pack's role and stage +# +# Writes work/pack-.txt (work/ ignores itself). An M/T pack is the role stub +# agents/-agent.md plus every baseline file routing_baseline_pack (bin/lib/skill-routing.sh) +# lists for and at , with bin/runbook-slice.sh expanded to its output, so the +# file list is the counted one. The ref is checked out in a temporary detached worktree, never in +# the caller's tree. Every pack ends with a PACK-END line; an agent that quotes it back read to +# the end (score.sh --reply). Prints arm, ref, routed words/files, and pack words. +set -eu +AB="$(cd "$(dirname "$0")" && pwd)"; REPO="$(cd "$AB/../.." && pwd)" +usage() { sed -n '4,8p' "$0" >&2; exit 2; } +ARM="${1:-}"; [ $# -gt 0 ] && shift +case "$ARM" in M|T) REF="${1:-}"; [ -n "$REF" ] || usage; shift ;; N) REF="" ;; *) usage ;; esac +ROLE=mdl; STAGE=5; DROP="" +while [ $# -gt 0 ]; do + case "$1" in + --without) DROP="${2:-}"; [ -n "$DROP" ] || usage ;; + --role) ROLE="${2:-}" ;; --stage) STAGE="${2:-}" ;; + *) usage ;; + esac + shift 2 || usage +done +mkdir -p "$AB/work"; [ -f "$AB/work/.gitignore" ] || printf '*\n' > "$AB/work/.gitignore" +OUT="$AB/work/pack-$ARM.txt"; BODY="$AB/work/.pack-$ARM.body" + +if [ "$ARM" = N ]; then + echo "You are an agent that writes Mendix models with mxcli, using MDL (Mendix Definition Language) scripts. You have general knowledge of Mendix and MDL. There are no project-specific rules or skills loaded for this session." > "$BODY" + info="N no pack (preamble only)" +else + TMPD="$(mktemp -d "${TMPDIR:-/tmp}/behaviour-pack.XXXXXX")"; WT="$TMPD/wt" + trap 'git -C "$REPO" worktree remove --force "$WT" >/dev/null 2>&1 || true; rm -rf "$TMPD"' EXIT + git -C "$REPO" worktree add --detach --quiet "$WT" "$REF" + line="$(cd "$WT" && bash -c '. bin/lib/skill-routing.sh; routing_baseline_pack "$1" "$PWD" "$2"' _ "$STAGE" "$ROLE")" + IFS=$'\t' read -r words files paths _ <<< "$line" + if [ -n "$DROP" ]; then + case " $paths " in *" $DROP "*) : ;; *) echo "build-packs: '$DROP' is not in the $ROLE Stage $STAGE pack at $REF" >&2; exit 2 ;; esac + fi + sha="$(git -C "$WT" rev-parse --short HEAD)" + { + echo "##### CONTEXT PACK — $ROLE role, Stage $STAGE (toolkit @ $sha)" + echo "##### This is everything you (the $ROLE agent) have preloaded. Toolkit root is referred to as /." + echo + echo "===== FILE: agents/$ROLE-agent.md (your role stub) =====" + cat "$WT/agents/$ROLE-agent.md" + for p in $paths; do + [ "$p" = "$DROP" ] && continue + echo + if [ "$p" = "bin/runbook-slice.sh" ]; then + echo "===== OUTPUT OF: bin/runbook-slice.sh $STAGE =====" + (cd "$WT" && bash bin/runbook-slice.sh "$STAGE") + else + echo "===== FILE: $p =====" + cat "$WT/$p" + fi + done + } > "$BODY" + info="$ARM $REF@$sha routing_baseline_pack=$words words/$files files" + [ -z "$DROP" ] || info="$info, without $DROP: $((words - $(LC_ALL=C wc -w < "$WT/$DROP"))) words/$((files - 1)) files" +fi +{ cat "$BODY"; echo; echo "===== END OF CONTEXT PACK =====" + printf 'PACK-END: %s-%s\n' "$ARM" "$(cksum < "$BODY" | cut -d' ' -f1)"; } > "$OUT" +rm -f "$BODY" +printf '%s\tpack=%s words, %s lines\n' "$info" "$(LC_ALL=C wc -w < "$OUT" | tr -d ' ')" "$(wc -l < "$OUT" | tr -d ' ')" diff --git a/tests/behaviour/fixtures/M-page-r5.md b/tests/behaviour/fixtures/M-page-r5.md new file mode 100644 index 00000000..25e8ddba --- /dev/null +++ b/tests/behaviour/fixtures/M-page-r5.md @@ -0,0 +1,274 @@ +### MDL + +```mdl +-- 10-orders-pages.mdl (Sales, feature group Orders) +-- Draft only: NOT checked, NOT executed. Owner of: Sales.Order_Overview, Sales.Order_NewEdit and the six +-- microflows below. Does not touch the domain model (02-sales-domain.mdl owns it). +-- Deliberate deviation from the request: NO "Last changed by" column. Binding a widget to the +-- AutoChangedBy member (changedBy) is STOP #4 in learned-mdl-preflight.md (check passes, mx check +-- gives CE1613). See the report notes for the decision the main session must take. +-- Forward reference: the microflows that show Order_Overview / Order_NewEdit are written before the +-- pages. Overview -> NewEdit -> Overview is a cycle, so one forward reference is unavoidable. +-- No @position anywhere (microflow-preflight.md, Layout). + +-- ===================== Microflows ===================== + +create or modify microflow Sales."NAV_Home" () +folder 'Orders/Microflows' +begin + show page Sales."Home"; +end; + +create or modify microflow Sales."NAV_Order_Overview" () +folder 'Orders/Microflows' +begin + show page Sales."Order_Overview"; +end; + +create or modify microflow Sales."ACT_Order_New" () +folder 'Orders/Microflows' +begin + $NewOrder = create Sales."Order" (); + show page Sales."Order_NewEdit"($Order = $NewOrder); +end; + +create or modify microflow Sales."ACT_Order_Edit" ($Order: Sales."Order") +folder 'Orders/Microflows' +begin + show page Sales."Order_NewEdit"($Order = $Order); +end; + +create or modify microflow Sales."ACT_Order_Save" ($Order: Sales."Order") +folder 'Orders/Microflows' +begin + commit $Order with events refresh; + close page; +end; + +create or modify microflow Sales."ACT_Order_Cancel" ($Order: Sales."Order") +folder 'Orders/Microflows' +begin + rollback $Order refresh; + close page; +end; + +grant execute on microflow Sales."NAV_Home" to Sales."User"; +grant execute on microflow Sales."NAV_Order_Overview" to Sales."User"; +grant execute on microflow Sales."ACT_Order_New" to Sales."User"; +grant execute on microflow Sales."ACT_Order_Edit" to Sales."User"; +grant execute on microflow Sales."ACT_Order_Save" to Sales."User"; +grant execute on microflow Sales."ACT_Order_Cancel" to Sales."User"; + +-- ===================== Page 1: Order_Overview ===================== + +create or replace page Sales."Order_Overview" ( + Title: 'Orders', + Layout: Atlas_Core.Atlas_Default, + Folder: 'Orders/Pages' +) +{ + container ctnPageHead (Class: 'page-header spacing-outer-bottom-large') { + container ctnCrumb (Class: 'breadcrumb') { + linkbutton lnkHome (Caption: 'Home', Class: 'breadcrumb-link', Action: MICROFLOW Sales."NAV_Home"()) + dynamictext txtCrumbSep (Content: ' › ') + dynamictext txtCrumbCurrent (Content: 'Orders', Class: 'breadcrumb-current') + } + dynamictext txtTitle (Content: 'Orders', RenderMode: H1, Class: 'page-title') + actionbutton btnNewOrder (Caption: 'New order', ButtonStyle: Primary, Action: MICROFLOW Sales."ACT_Order_New"()) + } + + container ctnOrders (Class: 'card spacing-outer-bottom-large') { + datagrid dgOrders (DataSource: database from Sales."Order" sort by "OrderDate" desc) { + column colOrderNumber (Attribute: "OrderNumber", Caption: 'Order number') + column colOrderDate (Attribute: "OrderDate", Caption: 'Order date') + column colStatus (Attribute: "Status", Caption: 'Status') + column colCustomer (Attribute: Order_Customer/Name, Caption: 'Customer') + column colTotal (Attribute: "TotalAmount", Caption: 'Total') + column colEdit (Caption: 'Actions', ShowContentAs: customContent) { + actionbutton btnEditOrder (Caption: 'Edit', Action: MICROFLOW Sales."ACT_Order_Edit"(Order: $currentObject)) + } + EmptyPlaceholder: { + container ctnEmpty (Class: 'empty-state') { + dynamictext txtEmpty (Content: 'No orders yet.') + } + } + } + } +} + +grant view on page Sales."Order_Overview" to Sales."User"; + +-- ===================== Page 2: Order_NewEdit ===================== + +create or replace page Sales."Order_NewEdit" ( + Params: { $Order: Sales."Order" }, + Title: 'Edit order', + Layout: Atlas_Core.Atlas_Default, + Folder: 'Orders/Pages' +) +{ + container ctnPageHead (Class: 'page-header spacing-outer-bottom-large') { + container ctnCrumb (Class: 'breadcrumb') { + linkbutton lnkHome (Caption: 'Home', Class: 'breadcrumb-link', Action: MICROFLOW Sales."NAV_Home"()) + dynamictext txtCrumbSep1 (Content: ' › ') + linkbutton lnkOrders (Caption: 'Orders', Class: 'breadcrumb-link', Action: MICROFLOW Sales."NAV_Order_Overview"()) + dynamictext txtCrumbSep2 (Content: ' › ') + dynamictext txtCrumbCurrent (Content: 'Edit order', Class: 'breadcrumb-current') + } + dynamictext txtTitle (Content: 'Edit order', RenderMode: H1, Class: 'page-title') + } + + container ctnForm (Class: 'card spacing-outer-bottom-large') { + dataview dvOrder (DataSource: $Order) { + textbox txtOrderNumber (Label: 'Order number', Attribute: "OrderNumber") + datepicker dpOrderDate (Label: 'Order date', Attribute: "OrderDate") + radiobuttons rbStatus (Label: 'Status', Attribute: "Status") + combobox cbCustomer (Label: 'Customer', Attribute: Order_Customer, DataSource: database from Sales."Customer" sort by "Name" asc, CaptionAttribute: "Name") + textarea taNote (Label: 'Note', Attribute: "Note") + footer ftOrder { + actionbutton btnSave (Caption: 'Save', ButtonStyle: Primary, Action: MICROFLOW Sales."ACT_Order_Save"(Order: $Order)) + actionbutton btnCancel (Caption: 'Cancel', Action: MICROFLOW Sales."ACT_Order_Cancel"(Order: $Order)) + } + } + } +} + +grant view on page Sales."Order_NewEdit" to Sales."User"; +``` + +### WRITE MODES + +- create or modify microflow Sales."NAV_Home" — CLI via ./bin/exec.sh (Studio Pro closed) — Step 0 default; plain show-page flow, no STOP row applies. +- create or modify microflow Sales."NAV_Order_Overview" — CLI via ./bin/exec.sh — same; carries a forward reference to the page created later in the script. +- create or modify microflow Sales."ACT_Order_New" — CLI via ./bin/exec.sh — create-object plus show-page, no association set, no STOP row. +- create or modify microflow Sales."ACT_Order_Edit" — CLI via ./bin/exec.sh — single show-page with a parameter. +- create or modify microflow Sales."ACT_Order_Save" — CLI via ./bin/exec.sh — commit with events plus refresh, then close page (page-triggered, so refresh applies). +- create or modify microflow Sales."ACT_Order_Cancel" — CLI via ./bin/exec.sh — rollback with refresh, then close page. +- grant execute on microflow Sales."NAV_Home" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- grant execute on microflow Sales."NAV_Order_Overview" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- grant execute on microflow Sales."ACT_Order_New" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- grant execute on microflow Sales."ACT_Order_Edit" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- grant execute on microflow Sales."ACT_Order_Save" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- grant execute on microflow Sales."ACT_Order_Cancel" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- create or replace page Sales."Order_Overview" — CLI via ./bin/exec.sh — grid has no conditional visible/editable inside customContent (STOP #1 not hit); association column written unquoted (STOP #26); the AutoChangedBy column is omitted (STOP #4). +- grant view on page Sales."Order_Overview" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. +- create or replace page Sales."Order_NewEdit" — CLI via ./bin/exec.sh — combobox in association mode is CLI-writable on v0.24+ (Step 0); no Editable or Visible expressions used. +- grant view on page Sales."Order_NewEdit" to Sales."User" — CLI via ./bin/exec.sh — co-located grant per the brief. + +### PREFLIGHT + +**Step 0 write mode.** Default CLI through `./bin/exec.sh`, Studio Pro closed. No operation needs Studio Pro by hand or MCP. Per-document folder, read off the brief's folder plan: pages go to `Orders/Pages`, microflows to `Orders/Microflows`. All 8 created documents carry a `folder` (6 microflows, 2 pages). No enumerations or constants are created, so no `move` statements are needed. + +**microflow-preflight** (one block per microflow, all Simple): +- `microflow-preflight: no trigger` — Sales.NAV_Home (linear, 1 activity, no loop). +- `microflow-preflight: no trigger` — Sales.NAV_Order_Overview (linear, 1 activity, no loop). +- `microflow-preflight: no trigger` — Sales.ACT_Order_New (tier Simple: 2 activities, no loop, one entity). +- `microflow-preflight: no trigger` — Sales.ACT_Order_Edit (tier Simple: 1 activity). +- `microflow-preflight: no trigger` — Sales.ACT_Order_Save (tier Simple: 2 activities; one commit, not in a loop; refresh used because page-triggered). +- `microflow-preflight: no trigger` — Sales.ACT_Order_Cancel (tier Simple: 2 activities). +- Layout: no `@position` anywhere. Annotations: none; no activity has a non-obvious why. + +**ui-preflight-pages (pages Order_Overview, Order_NewEdit):** +- Step 0, entry points: `mxcli impact` / `callers` / `DESCRIBE PAGE Sales."Home"` — NOT RUN, no tools or model available. Known from the prompt: `Sales."Home"` exists. Both new pages are reached only through the breadcrumb and button microflows written here. No navigation item is added; the brief requests none. +- Step 1, wireframes: `design/wireframes/order-overview.html` and `order-newedit.html` — NOT READ as files (file access not allowed); worked from the transcription in the task. Binding table taken from that transcription. +- Step 2, design system: `design/ds.css` not opened. Only the classes the task lists are used: `page-header`, `page-title`, `breadcrumb`, `breadcrumb-link`, `breadcrumb-current`, `card`, `empty-state`. The only other classes are the Atlas spacing utility `spacing-outer-bottom-large` (the section-rhythm class from design-spacing.md §2). No inline styles. No invented class. +- Step 2, hard rule B0 (`project-bin/check-design-reaches-app.sh`, `docs/design-reaches-app.md`) — NOT RUN, no build or tools. Binding UNPROVEN. +- Page column: the task states no wireframe width cap and `ds.css` lists no page-column class. Gap flagged: if the wireframes cap `main`, a `.page-column` class must be promoted into `main.scss` and added to the page body containers. Not done here, because `main.scss` is out of scope. +- Step 3, gallery: no StyleGallery exists, so nothing to reuse. Fallback is the design-system classes above. The empty state uses the `empty-state` class. Status renders as the plain enum column: no badge component exists to reuse, and none was reimplemented. +- Step 4, cross-check: + - Bindings match the task's column and field list, with `Last changed by` omitted (STOP #4). + - Block separation: each section sits in a `card`. + - Page scaffold: `ctnPageHead` has a breadcrumb plus exactly one `RenderMode: H1` on each page. + - Spacing: every top-level section carries `spacing-outer-bottom-large`. + - Empty state: the grid has one. + - Response after a write: Save commits then closes the page, Cancel rolls back then closes the page, and New/Edit navigate. + - Validation feedback: no required or unique fields were specified, so none is added. No VAL_ microflow was requested, so none is invented. + - Conditional visibility: none used. + - Class carrier: classes sit on CONTAINER, DYNAMICTEXT and LINKBUTTON. `breadcrumb-link` on a LINKBUTTON follows the task's class list, and its carrier is unverified because ds.css was not read. + - Computed class names: no DynamicClasses. +- Accessibility floor: one H1 per page, visible labels on all inputs, no images. + +**STOP-table check (learned-mdl-preflight.md), every row against the planned operations:** +- #1 (visible/editable inside customContent): not hit. The only customContent column holds one button with no expression. +- #1b (`Editable:`): not used. +- **#4 (AutoChangedBy binding): HIT. The requested "Last changed by" column bound to system `changedBy` is not written. STOP flagged.** Alternative per the row: omit it, or bind `AutoChangedDate`. The date column was not added unprompted. See the notes. +- #12 (ContentParams): all dynamictext content is quoted literals, none bound. +- #14 (narrowing a grant): not applicable, only new grants are written. +- #16 (CurrentUser XPath): not applicable. +- #18 (workflow): not applicable. +- #21 (DROP+CREATE of a shared document): no `drop`. All six microflows and both pages are new and owned by this script. +- #22 (idempotence): every create uses `create or modify` (microflows) or `create or replace` (pages). No bare `alter entity ... add attribute`, no bare `CREATE ASSOCIATION`. +- #23 (`mxcli diff -p .mpr 10-orders-pages.mdl --format struct`): NOT RUN, no tool access. Must run before exec to confirm none of the six microflow or two page names already exist. +- #24 (DesignProperties): none used. +- #25 (loop, more than 20 activities, list from list): not applicable. +- #26 (DG2 association column): written unquoted as `Order_Customer/Name`. The module-qualified form was not used because the row's example is the unqualified form; unverified. +- #28 (changing a microflow near a return): not applicable, all microflows are new. +- Gotchas: + - `refresh` is placed at the end of the statement. + - GRANT roles are module-qualified. + - No `elseif`. + - No `mdl 1;` header. + - Unquoted association binding on the combobox is the same convention as #26. + +**Final self-check re-read** against the table: no conditional visibility in customContent, the one association path is unquoted, no unprefixed association steps in expressions (there are none), no `Editable:`. Done by reading only. `mxcli check --references` — NOT RUN. + +**UI cross-reference block:** + +``` +UI cross-reference: + Wireframe: design/wireframes/order-overview.html, order-newedit.html [NOT READ as files - transcription in the task used] + Design system: design/ds.css [NOT OPENED] - classes used: page-header, page-title, breadcrumb, breadcrumb-link, breadcrumb-current, card, empty-state, spacing-outer-bottom-large (Atlas utility) + Gallery example: none - no StyleGallery exists + Components reused: none needed + Reuse skipped: none (no gallery component exists) + Empty states: yes - dgOrders has an empty-state container reading "No orders yet." + Gaps / MCP fallbacks: STOP #4 - Last changed by column not built. Page column class unknown. No MCP fallback needed. + Shell check: bin/check-page-shell.sh - NOT RUN, no tool access + Fidelity score: page-fidelity.js - NOT RUN, no tool access (docs/PAGE-FIDELITY.tsv not written) + Class promotion: self-audit diff - NOT RUN, no tool access +``` + +### REPORT + +```report +scripts: 1 +operations: 16 +studio_pro_ops: 0 +stops_flagged: 1 +checks_not_run: 9 +open_questions: 1 +``` + +Notes for the main session: + +- **Script:** `mdlsource/10-orders-pages.mdl` creates 6 microflows (`NAV_Home`, `NAV_Order_Overview`, `ACT_Order_New`, `ACT_Order_Edit`, `ACT_Order_Save`, `ACT_Order_Cancel`) and 2 pages (`Order_Overview`, `Order_NewEdit`). The 6 microflows carry `folder 'Orders/Microflows'` and the 2 pages `Folder: 'Orders/Pages'`. Each is followed by its grant to `Sales."User"`. Operations: 8 creates and 8 grants. Nothing needs Studio Pro or MCP. +- **Open question and STOP #4 (for the product owner):** the "Last changed by" column was not built. Binding a widget to the system `changedBy` member passes `mxcli check` and then fails `mx check` with CE1613. Options: + 1. Ship without the column. This is what the script does. + 2. Show `changedDate` ("Last changed") instead, which binds cleanly. + 3. Add a stored String attribute, for example `"LastChangedByName"`, set in `ACT_Order_Save` from `$currentUser/Name`. This needs a domain-script change. + My recommendation is option 3 if the user's name is the requirement, otherwise option 2. The brief listed no open questions, so this is new, and the module brief and BRD should be updated once the owner decides. +- **Checks not run (9):** + 1. `mxcli check --references`. + 2. `mxcli diff --format struct` (STOP #23). + 3. `check-page-shell.sh`. + 4. `page-fidelity.js`. + 5. The class-promotion diff. + 6. `check-design-reaches-app.sh`. + 7. The Step 0 `impact` / `callers` / `DESCRIBE PAGE`. + 8. The dispatch log line (`.claude/loop/dispatch/Sales/scripts.tsv`, which cannot be written without a write tool outside the output path). + 9. Reading the wireframe, ds.css and brief files. + The exec is also not run, as instructed. +- **Unverified syntax risks.** Written from the skills and CLAUDE.md summaries; each needs `mxcli check` before exec. + - The microflow `folder '...'` placement before `begin`. + - Page-level `Params: { ... }`. + - The `EmptyPlaceholder: { ... }` block on the data grid. + - The `database from ... sort by ... desc` data-grid source. + - `ShowContentAs: customContent` with a nested action button. + - Combobox `DataSource` and `CaptionAttribute`. + - `DataSource: $Order` on the data view. + - `$currentObject` as the microflow argument inside the row button. + - The association column path `Order_Customer/Name`. + - Status as a plain radio group bound to the enumeration. +- **Forward reference.** `NAV_Order_Overview` and `ACT_Order_New` / `ACT_Order_Edit` show pages created later in the same script. `Overview -> NewEdit -> Overview` is a cycle, so one forward reference cannot be avoided. If `exec` refuses it, split the script and use a `Stub:` page for the first, scored with `--stub`. +- **Behaviour choices.** New order: `ACT_Order_New` creates an uncommitted object and Save commits it. Cancel rolls it back, which discards a new object. No validation microflow was added, because no required or unique fields were specified. +- **Page column.** Possible gap, see PREFLIGHT. diff --git a/tests/behaviour/fixtures/N-mf-r1.md b/tests/behaviour/fixtures/N-mf-r1.md new file mode 100644 index 00000000..47731741 --- /dev/null +++ b/tests/behaviour/fixtures/N-mf-r1.md @@ -0,0 +1,94 @@ +### MDL + +```mdl +-- mdlsource/11-orders-submit.mdl +-- Draft only. Not executed. + +/** + * Submits every order in the list that has a customer and a positive total. + * Orders failing the check get validation feedback on TotalAmount. + * + * @param $OrderList Orders selected for submission + * @returns true when every order in the list was submitted + */ +CREATE OR REPLACE MICROFLOW Sales."SUB_Orders_SubmitReady" ( + $OrderList: List of Sales."Order" +) +RETURNS Boolean AS $AllSubmitted +FOLDER 'Orders/Microflows' +BEGIN + DECLARE $AllSubmitted Boolean = true; + + LOOP $Order IN $OrderList + BEGIN + IF $Order/Sales."Order_Customer" = empty OR $Order/"TotalAmount" <= 0 THEN + VALIDATION FEEDBACK $Order/"TotalAmount" MESSAGE 'Order cannot be submitted: customer and a positive total are required.'; + SET $AllSubmitted = false; + ELSE + CHANGE $Order ("Status" = Sales."ENUM_OrderStatus".Submitted); + COMMIT $Order; + END IF; + END LOOP; + + RETURN $AllSubmitted; +END; +/ + +/** + * Nightly maintenance chain: purge old drafts, recalculate totals, + * then submit all remaining Draft orders that are ready. + */ +CREATE OR REPLACE MICROFLOW Sales."SUB_Nightly_Maintenance" () +BEGIN + CALL MICROFLOW Sales."SUB_PurgeOldDrafts" (); + CALL MICROFLOW Sales."SUB_RecalcTotals" (); + + DECLARE $DraftOrders List of Sales."Order" = empty; + RETRIEVE $DraftOrders FROM Sales."Order" + WHERE "Status" = Sales."ENUM_OrderStatus".Draft; + + DECLARE $AllSubmitted Boolean = false; + SET $AllSubmitted = CALL MICROFLOW Sales."SUB_Orders_SubmitReady" ($OrderList = $DraftOrders); +END; +/ + +GRANT EXECUTE ON MICROFLOW Sales."SUB_Orders_SubmitReady" TO Sales."User"; +GRANT EXECUTE ON MICROFLOW Sales."SUB_Nightly_Maintenance" TO Sales."User"; +``` + +### WRITE MODES + +- create or replace microflow Sales."SUB_Orders_SubmitReady" — ./bin/exec.sh (Studio Pro closed) — no inline association set; the association is only read in an `= empty` test +- create or replace microflow Sales."SUB_Nightly_Maintenance" — ./bin/exec.sh (Studio Pro closed) — plain calls plus a retrieve; no association writes +- grant execute on microflow Sales."SUB_Orders_SubmitReady" to Sales."User" — ./bin/exec.sh (Studio Pro closed) — access rule per the brief +- grant execute on microflow Sales."SUB_Nightly_Maintenance" to Sales."User" — ./bin/exec.sh (Studio Pro closed) — re-asserted because replacing the microflow may drop its grants + +### PREFLIGHT + +No project skills or preflight checklists were loaded for this session. I therefore have no skill-mandated checklist to post and no STOP table to check against. NOT RUN — no preflight skill, STOP table, or checklist was provided. + +The following are my own checks, not a substitute for the STOP table: +- Syntax check (`./mxcli check`): NOT RUN — cannot run tools here. +- Reference check (`check --references`): NOT RUN — cannot run tools here. +- mxbuild / `mx check`: NOT RUN — cannot run tools here. +- Quoting: every identifier I wrote is double-quoted, per the project convention. +- Loop size: a few hundred orders with one commit per iteration is acceptable. Collecting the orders and committing the list once after the loop would be faster. I kept per-order commits so that a failure only affects one order. + +Risks to resolve before exec: +1. `SUB_Nightly_Maintenance` is also edited by `mdlsource/04-maintenance.mdl`. A `create or replace` here overwrites those edits, and the result depends on script order. I could not see that script, so I do not know what it changes. +2. I assumed the existing signature has no parameters and no return value, and I restated the body from the description alone. The existing folder, documentation, annotations, error handling and any other grants are unknown and may be lost. `DESCRIBE MICROFLOW Sales."SUB_Nightly_Maintenance"` should be run first. I could not run it. +3. The folder for `SUB_Nightly_Maintenance` is not restated, because I do not know it. If it lives in a folder, replacing it may move it to the module root. +4. Behaviour: the nightly job now submits every Draft order that passes validation. Validation feedback raised in a non-UI context is only logged and does not stop the job. This is worth confirming with the lead. + +### REPORT + +```report +scripts: 1 +operations: 4 +studio_pro_ops: 0 +stops_flagged: 0 +checks_not_run: 3 +open_questions: 3 +``` + +Notes: `checks_not_run` counts the syntax, reference and mxbuild checks. The open questions are the conflict with `04-maintenance.mdl`, the unknown existing signature and folder of `SUB_Nightly_Maintenance`, and whether the nightly job should auto-submit Draft orders. `stops_flagged` is 0 only because no STOP table was available to check against. diff --git a/tests/behaviour/fixtures/N-page-r1.md b/tests/behaviour/fixtures/N-page-r1.md new file mode 100644 index 00000000..520a1b2c --- /dev/null +++ b/tests/behaviour/fixtures/N-page-r1.md @@ -0,0 +1,108 @@ +### MDL + +```mdl +-- 10-orders-pages.mdl (DRAFT, not executed). Syntax drafted from general knowledge; no skill text loaded. + +CREATE OR REPLACE PAGE Sales."Order_Overview" +( + Title: 'Orders', + Layout: Atlas_Core.Atlas_Default, + Folder: 'Orders/Pages' +) +{ + CONTAINER cntHeader (Class: 'page-header') { + CONTAINER cntBreadcrumb (Class: 'breadcrumb') { + ACTIONBUTTON btnCrumbHome (Caption: 'Home', Class: 'breadcrumb-link', Action: PAGE Sales."Home") + DYNAMICTEXT txtCrumbOrders (Content: 'Orders', Class: 'breadcrumb-current') + } + DYNAMICTEXT txtTitle (Content: 'Orders', RenderMode: H1, Class: 'page-title') + ACTIONBUTTON btnNew (Caption: 'New order', ButtonStyle: Primary, Action: PAGE Sales."Order_NewEdit" (NEW Sales."Order")) + } + + DATAGRID dgOrders (DataSource: DATABASE Sales."Order" SORT BY "OrderDate" DESC) + { + COLUMN colNumber (Caption: 'Order number', Attribute: "OrderNumber") + COLUMN colDate (Caption: 'Order date', Attribute: "OrderDate") + COLUMN colStatus (Caption: 'Status', Attribute: "Status") + COLUMN colCustomer (Caption: 'Customer', Attribute: Sales."Order_Customer"/Sales."Customer"/"Name") + COLUMN colTotal (Caption: 'Total', Attribute: "TotalAmount") + -- UNVERIFIED: system member changedBy (association to System.User); path form not checked + COLUMN colChangedBy (Caption: 'Last changed by', Attribute: System."changedBy"/System."User"/"Name") + COLUMN colActions (Caption: 'Actions') { + ACTIONBUTTON btnEdit (Caption: 'Edit', Action: PAGE Sales."Order_NewEdit" ($Order: $currentObject)) + } + } + + -- Empty state: UNVERIFIED mechanism (no verified grid empty-text property known) + CONTAINER cntEmpty (Class: 'empty-state', Visible: 'count($dgOrders) = 0') { + DYNAMICTEXT txtEmpty (Content: 'No orders yet.') + } +}; + +CREATE OR REPLACE PAGE Sales."Order_NewEdit" +( + Title: 'Edit order', + Layout: Atlas_Core.Atlas_Default, + Folder: 'Orders/Pages', + Params: { $Order: Sales."Order" } +) +{ + CONTAINER cntHeader (Class: 'page-header') { + CONTAINER cntBreadcrumb (Class: 'breadcrumb') { + ACTIONBUTTON btnCrumbHome (Caption: 'Home', Class: 'breadcrumb-link', Action: PAGE Sales."Home") + ACTIONBUTTON btnCrumbOrders (Caption: 'Orders', Class: 'breadcrumb-link', Action: PAGE Sales."Order_Overview") + DYNAMICTEXT txtCrumbEdit (Content: 'Edit order', Class: 'breadcrumb-current') + } + DYNAMICTEXT txtTitle (Content: 'Edit order', RenderMode: H1, Class: 'page-title') + } + + DATAVIEW dvOrder (DataSource: $Order, Class: 'card') { + TEXTBOX txtOrderNumber (Label: 'Order number', Attribute: "OrderNumber") + DATEPICKER dpOrderDate (Label: 'Order date', Attribute: "OrderDate") + RADIOBUTTONS rbStatus (Label: 'Status', Attribute: "Status") + COMBOBOX cbCustomer (Label: 'Customer', Attribute: Sales."Order_Customer", CaptionAttribute: "Name") + TEXTAREA taNote (Label: 'Note', Attribute: "Note") + FOOTER { + ACTIONBUTTON btnSave (Caption: 'Save', ButtonStyle: Primary, Action: SAVE_CHANGES) + ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES) + } + } +}; + +GRANT VIEW ON PAGE Sales."Order_Overview" TO Sales."User"; +GRANT VIEW ON PAGE Sales."Order_NewEdit" TO Sales."User"; +``` + +### WRITE MODES + +- create page Sales."Order_Overview" — `./bin/exec.sh mdlsource/10-orders-pages.mdl` — plain page, no custom widget config; Studio Pro closed. +- create page Sales."Order_NewEdit" — `./bin/exec.sh` — the combobox is in association mode (Order_Customer), which in the project's write-mode table needs hand-rolled MCP (`pg_patch_page`) with Studio Pro open. This page must be split out or the combobox patched via MCP. +- datagrid column over cross-module path System.changedBy/User/Name — MCP/Studio Pro — cross-module datasource traversal is a hand-rolled MCP case in the project table; also unverified. +- grant view on pages — `./bin/exec.sh` — plain access grant, Studio Pro closed. + +### PREFLIGHT + +No project skills or preflight files were loaded in this session (the pack contains only a generic role line), so I have no checklist to post. Skill-level "no trigger" lines: NOT APPLICABLE, nothing loaded. + +STOP-table check: NOT RUN — `learned-mdl-preflight.md` was not in my context and I may not open files. I cannot claim the script clears it. + +Checks not run: `mxcli check` syntax, `check --references`, mx check, mxbuild, and a capability probe of the v0.26.0 binary. All NOT RUN — I cannot execute tools. + +Items I could not verify from context: +1. Page and widget property syntax (Folder, Params, SORT BY, Visible expression, FOOTER, SAVE_CHANGES/CANCEL_CHANGES) is drafted from general MDL knowledge. +2. "Last changed by": `changedBy` is an association to System.User, not a plain attribute. The path syntax is a guess. Showing the user name also needs Sales."User" read access to System.User (not granted in this slice). +3. Empty state: I know of no verified datagrid empty-text property; the visibility expression on a sibling container is a guess. +4. Identifiers are quoted per project convention; the unquoted forms Sales."Order_Customer"/System."changedBy" are used as qualified names. + +### REPORT + +```report +scripts: 1 +operations: 4 +studio_pro_ops: 2 +stops_flagged: 3 +checks_not_run: 5 +open_questions: 3 +``` + +Notes: Open questions: (1) correct MDL path for the changedBy column and the System.User read access it needs; (2) mechanism for the grid empty state; (3) whether the association-mode combobox goes via MCP or is split into a separate script. Nothing was executed. stops_flagged counts the changedBy traversal, the association combobox and the unverified empty state. diff --git a/tests/behaviour/fixtures/T-mf-r5.md b/tests/behaviour/fixtures/T-mf-r5.md new file mode 100644 index 00000000..1e1bd9ae --- /dev/null +++ b/tests/behaviour/fixtures/T-mf-r5.md @@ -0,0 +1,135 @@ +### MDL + +```mdl +-- 11-orders-submit.mdl (Sales / Orders feature) -- DRAFT, never exec'd by the drafter +-- +-- Owns: Sales."SUB_Orders_SubmitReady" (new). +-- Restates (STOP #21, shared document): Sales."SUB_Nightly_Maintenance". The restatement below is +-- written from the body the lead described, NOT from a live describe (NOT RUN, no model access). +-- Call sites preserved, in order: Sales."SUB_PurgeOldDrafts", then Sales."SUB_RecalcTotals". +-- Call site added: Sales."SUB_Orders_SubmitReady", last. +-- mdlsource/04-maintenance.mdl also edits SUB_Nightly_Maintenance; do not exec this script until the +-- owner question in the report is settled and `describe microflow Sales.SUB_Nightly_Maintenance` +-- has been diffed against the second statement. + +/** + * Submits every order in the list that has a customer and a positive total. + * Orders that fail either rule get validation feedback on TotalAmount and stay unchanged. + * + * @param $OrderList The orders to submit (multi-select grid, up to a few hundred) + * @returns true when every order in the list was submitted + */ +create or modify microflow Sales."SUB_Orders_SubmitReady" ( + $OrderList: List of Sales."Order" +) +returns Boolean as $AllSubmitted +folder 'Orders/Microflows' +begin + declare $AllSubmitted Boolean = true; + $Order_CommitList = create list of Sales."Order"; + + loop $Order in $OrderList + begin + if $Order/Sales.Order_Customer = empty or $Order/TotalAmount <= 0 then + validation feedback $Order/TotalAmount message 'Order cannot be submitted: customer and a positive total are required.'; + set $AllSubmitted = false; + else + change $Order ("Status" = Sales."ENUM_OrderStatus".Submitted); + add $Order to $Order_CommitList; + end if; + end loop; + + @annotation 'One commit after the loop: orders are collected in the loop and committed together (no commit per item).' + commit $Order_CommitList with events refresh; + + return $AllSubmitted; +end; +/ + +grant execute on microflow Sales."SUB_Orders_SubmitReady" to Sales."User"; + +/** + * Nightly maintenance chain: purge old drafts, recalculate totals, then submit the ready draft orders. + */ +create or replace microflow Sales."SUB_Nightly_Maintenance" () +folder '<>' +begin + call microflow Sales."SUB_PurgeOldDrafts"(); + call microflow Sales."SUB_RecalcTotals"(); + + retrieve $DraftOrders from Sales."Order" + where "Status" = Sales."ENUM_OrderStatus".Draft; + + $AllSubmitted = call microflow Sales."SUB_Orders_SubmitReady"(OrderList = $DraftOrders); +end; +/ +``` + +### WRITE MODES + +- create or modify microflow Sales."SUB_Orders_SubmitReady" (folder Orders/Microflows) — CLI via `./bin/exec.sh`, Studio Pro closed — microflow with no inline association set; the association is only read (`= empty`), nothing in the STOP table applies to the write. +- grant execute on microflow Sales."SUB_Orders_SubmitReady" to Sales."User" — CLI via `./bin/exec.sh`, same script — grant is co-located with the element, per the brief's access table. +- create or replace microflow Sales."SUB_Nightly_Maintenance" — CLI via `./bin/exec.sh`, but BLOCKED on STOP #21/#23 checks (describe, diff, ownership) — mxcli has no ALTER MICROFLOW for an existing body, so whole-document restatement is the only write; the existing grants and folder must be read from the live model first. + +No operation needs Studio Pro. No enumeration or constant is created, so no `move` statement is owed. + +### PREFLIGHT + +**Step 0, write mode.** Default CLI through `bin/exec.sh` for all three operations (see WRITE MODES). Folder decided per document: SubmitReady from the brief's folder plan (`Orders/Microflows`). SUB_Nightly_Maintenance has no row in the brief's folder plan (the plan names feature group Orders only, and Nightly is maintenance, not an order feature), so I did not invent one: the statement carries a deliberate placeholder that fails `mxcli check` until the folder of the existing document is read off `describe` (or the architect names one). I did not fall through to module root. + +**microflow-preflight: Sales.SUB_Orders_SubmitReady — tier: Guided** +- Loops: 1. Per loop: retrieve n · commit/delete n (collected) · call n · batch strategy: none needed, one commit after the loop · expected list size: up to a few hundred +- Retrieves inside loops: 0 +- Commits/deletes inside loops: 0 — collected into Order_CommitList, one commit after the loop +- REST/Java/web-service calls inside loops: 0 +- Nested loops: 0 +- Lists built from lists: 1 — the commit list is built by `add` in the loop on purpose (it is the CommitList pattern from the skill, not a filter-able list operation, because each item is changed and conditionally included) +- Activities: top-level T + all loop bodies B = 5 + 5 = 10 (≤ 20, no SUB_ split needed) +- Top-level activities: 5 (declare, create list, loop, commit, return) (≤ 15) +- Decisions + loops: 2 (≤ 9) +- Layout: no @position + +**microflow-preflight: Sales.SUB_Nightly_Maintenance — no trigger** (no loop, 4 top-level activities, one retrieve of Order that is not committed in this flow; the commit happens inside the called SUB_). Size note: the retrieve of all Draft orders is unbounded; the expected volume is not stated, so see open question 3. + +**Validation-feedback rule (learned-microflow-patterns.md):** feedback lives in a SUB_ that returns a verdict, attribute named, collect-all (no `return` inside the loop, no `else` chaining past the failure), no `log error`, no annotation on it. The caller (ACT_ or the nightly flow) branches on the Boolean. + +**STOP-table check (learned-mdl-preflight.md), every row against the planned operations:** +- #21 (create or replace on a document another script also edits): TRIGGERED. SUB_Nightly_Maintenance is also edited by `mdlsource/04-maintenance.mdl`. Required: `describe microflow Sales.SUB_Nightly_Maintenance` first, diff my restatement against it, name preserved call sites in the header. NOT RUN (no model access). Header names the two call sites the lead described; any other call site (including one 04 adds) would be lost. Also breaks the one-owner-per-document rule (#22): two scripts own one microflow, so whichever runs last decides its body. +- #22 (non-idempotent statement): checked. The new microflow uses `create or modify`; the Nightly statement is `create or replace` (idempotent) as the lead asked. No bare CREATE. +- #23 (create or replace on a name that may exist): TRIGGERED for Nightly. `mxcli diff -p .mpr mdlsource/11-orders-submit.mdl --format struct` must be run and every `-` line read before exec. NOT RUN. +- #25 (loop in a microflow): TRIGGERED for SubmitReady; discharged by the checklist above. +- #28 (change near a return in a dense flow): not applicable, whole-body restatement, no insert before a return. +- #1, #1b, #4, #12, #24, #26 (page and widget rows): not applicable, no page or widget in this script. +- #18 (workflow): not applicable. +- #14, #16: not applicable, no entity grants are narrowed, no access-rule XPath. +- Other rows are retired. +- Gotchas checked: association path in the expression is the bare `= empty` test (allowed without the long form); `validation feedback` names the attribute (CE0091); enum value qualified as `Sales."ENUM_OrderStatus".Submitted`, no enum-to-String use; grant role is module-qualified (`Sales."User"`); no `[%CurrentUser%]` (the flow is also reachable from a nightly/system session); no `mdl 1;` header; no `@position`. +- Other preloaded rules applied: security-is-not-a-later-script (entity access for Sales."User" already granted by the domain script, taken from the prompt, not verified with SHOW SECURITY MATRIX: NOT RUN); degrade-to-judgement (missing live model named in each NOT RUN, verdicts still given). + +**Final self-check (re-read against the table):** no conditional visibility in a grid, no quoted DataGrid association paths, no unprefixed association step beyond the bare `= empty`, no quoted `Editable:`. Clean on re-read. + +**Page checklist:** `ui-preflight-pages.md`: no trigger, this script creates no page or snippet. + +### REPORT + +```report +scripts: 1 +operations: 3 +studio_pro_ops: 0 +stops_flagged: 3 +checks_not_run: 5 +open_questions: 3 +``` + +Notes. + +- stops_flagged counts STOP #21, #23 (both on the Nightly restatement, blocking exec until the live describe and diff are read) and #25 (SubmitReady loop, discharged by the posted checklist). +- checks_not_run (all because I may not run tools): (1) `mxcli check mdlsource/11-orders-submit.mdl -p --references`; (2) `mxcli diff ... --format struct` (rule 23); (3) `describe microflow Sales.SUB_Nightly_Maintenance` and the diff of my restatement against it (rule 21); (4) `show access on microflow Sales.SUB_Nightly_Maintenance`, to re-grant whatever it lists after the replace; (5) the step 8a dispatch log line in `.claude/loop/dispatch/Sales/scripts.tsv` (script path, date, check result) — the check result cannot be recorded because the check did not run, so the main session should log it after the check. Also not done: post-exec `describe microflow Sales.SUB_Orders_SubmitReady` read-back (belongs to the main session after exec). I did not read the Wiring block or the brief from disk; I used the brief excerpt in the prompt. +- Open questions: + 1. Who owns SUB_Nightly_Maintenance, this script or `04-maintenance.mdl`? Two scripts owning one document means the later one silently overwrites the other. Recommend the edit move into `04-maintenance.mdl` (or 04 stops touching it), and then 11 only creates SubmitReady. + 2. Folder for SUB_Nightly_Maintenance: no row in the brief's folder plan and not an Orders-feature document. The statement holds a deliberate placeholder (it fails `check` on purpose). Needs the existing folder from `describe`, or an architect answer. I did not guess `Orders/Microflows`. + 3. The current Nightly signature (parameters, return type) and any call sites beyond the two described are unverified; I assumed no parameters and no return. Also unbounded retrieve of all Draft orders: confirm the nightly volume or add a limit/batch. +- Unverified-syntax risks (nothing was run): placement of `folder '...'` between the `returns ... as $x` clause and `begin`; `where "Status" = Sales."ENUM_OrderStatus".Draft` as an enum comparison in the retrieve (if `check` rejects it, retry unquoted, or compare as a string literal); `$Order/Sales.Order_Customer` written unquoted (paths stay unquoted per the skills, against the project's quote-everything convention; retry the quoted form if `check` objects); `call microflow ... (OrderList = $DraftOrders)` with the parameter name unquoted as in the skill examples. +- Decision to confirm: `commit ... with events refresh` — `refresh` is right for the grid caller but is meaningless in the nightly path (no client). Harmless; kept because the shared SUB_ is mostly page-called. The nightly path will also produce validation feedback with nobody to read it; the returned Boolean is assigned and ignored there. +- The empty list returns true (nothing failed). A null `TotalAmount` is not treated as invalid beyond what `<= 0` does; add `or $Order/TotalAmount = empty` if the model allows empty amounts. +- Folder per created document: `Sales."SUB_Orders_SubmitReady"` goes to `Orders/Microflows`; `Sales."SUB_Nightly_Maintenance"` is unresolved (open question 2). Sequencing: the main session should do the describe/diff/show-access for Nightly before exec, and consider splitting the Nightly restatement into its own script so a failure there cannot hide the SubmitReady write. diff --git a/tests/behaviour/results/2026-10-15-no-pack.tsv b/tests/behaviour/results/2026-10-15-no-pack.tsv new file mode 100644 index 00000000..50b9c88b --- /dev/null +++ b/tests/behaviour/results/2026-10-15-no-pack.tsv @@ -0,0 +1,12 @@ +# 2026-10-15 no-pack arm N: 35-word preamble only (build-packs.sh N), same tasks and model as the stub-diet run. tool_calls=4 every rep; tokens 76.5k-78.3k per rep (per-rep values not recorded). Caveat: every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'. +# Headline: page trap (bind AutoChangedBy) refused 0/5 here against 5/5 with either pack; mf passes mxcli check 1/5 against 5/5. +N-page-r1 arm=N task=page sections=4/4 report=1 wmode=4/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=0 crumbs=0/3 txtcrumb=0 h1=2/2 empty=1 uixref=0 read_all=1 tool_calls=4 tokens=NA +N-page-r2 arm=N task=page sections=4/4 report=1 wmode=4/4 quoted=100 folder=0/2 scan=0 check=5 planted=0 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=0 read_all=1 tool_calls=4 tokens=NA +N-page-r3 arm=N task=page sections=4/4 report=1 wmode=4/4 quoted=100 folder=0/2 scan=0 check=4 planted=0 crumbs=3/3 txtcrumb=0 h1=2/2 empty=0 uixref=0 read_all=1 tool_calls=4 tokens=NA +N-page-r4 arm=N task=page sections=4/4 report=1 wmode=4/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=0 crumbs=0/3 txtcrumb=0 h1=2/2 empty=1 uixref=0 read_all=1 tool_calls=4 tokens=NA +N-page-r5 arm=N task=page sections=4/4 report=1 wmode=3/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=0 crumbs=0/3 txtcrumb=0 h1=2/2 empty=1 uixref=0 read_all=1 tool_calls=4 tokens=NA +N-mf-r1 arm=N task=mf sections=4/4 report=1 wmode=4/4 quoted=100 folder=1/2 scan=0 check=PARSE planted=1 mfpre=0 cinloop=1 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=4 tokens=NA +N-mf-r2 arm=N task=mf sections=4/4 report=1 wmode=4/4 quoted=100 folder=2/2 scan=0 check=3 planted=1 mfpre=0 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=4 tokens=NA +N-mf-r3 arm=N task=mf sections=4/4 report=1 wmode=4/4 quoted=100 folder=2/2 scan=0 check=0 planted=1 mfpre=0 cinloop=1 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=4 tokens=NA +N-mf-r4 arm=N task=mf sections=4/4 report=1 wmode=3/3 quoted=100 folder=2/2 scan=0 check=1 planted=1 mfpre=0 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=4 tokens=NA +N-mf-r5 arm=N task=mf sections=4/4 report=1 wmode=4/4 quoted=100 folder=2/2 scan=0 check=4 planted=0 mfpre=0 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=4 tokens=NA diff --git a/tests/behaviour/results/2026-10-15-stub-diet.tsv b/tests/behaviour/results/2026-10-15-stub-diet.tsv new file mode 100644 index 00000000..12d9658c --- /dev/null +++ b/tests/behaviour/results/2026-10-15-stub-diet.tsv @@ -0,0 +1,45 @@ +# 2026-10-15 stub-diet A/B (#369/#378/#380). Arm M = master d16f578, arm T = d16f578 + #369 + #378 + #380 (local merge 833392c). One model fixed for every rep. +# read_all=0 rows: agent stopped reading the 4,822-line pack early; re-run, counted here, excluded from the comparison. tool_calls/tokens per rep were not recorded; ranges: M 15-24 calls ~245k tokens, T 7-14 calls 160-175k tokens. +# Findings: 39 reps read the whole pack (20 drafts, 19 probes), 3 partial reads, all on the 4,822-line M pack. T held the eight metrics that run's rule named (planted, scan, check, mfpre, cinloop, rinloop, uixref, crumbs); quoted on mf was lower (median 93 against 95), which the every-metric rule now reports. M ran one probe rep each, none valid for skills-over-scripts. +M-mf-r1 arm=M task=mf sections=4/4 report=1 wmode=4/3 quoted=94 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +M-mf-r2 arm=M task=mf sections=4/4 report=1 wmode=3/2 quoted=95 folder=1/1 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +M-mf-r3 arm=M task=mf sections=4/4 report=1 wmode=4/3 quoted=95 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +M-mf-r4 arm=M task=mf sections=4/4 report=1 wmode=3/3 quoted=64 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +M-mf-r5 arm=M task=mf sections=4/4 report=1 wmode=3/2 quoted=95 folder=1/1 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +M-page-r1 arm=M task=page sections=4/4 report=1 wmode=6/5 quoted=100 folder=0/3 scan=0 check=0 planted=1 crumbs=3/3 txtcrumb=0 h1=3/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +M-page-r2 arm=M task=page sections=4/4 report=1 wmode=6/5 quoted=100 folder=0/3 scan=0 check=PARSE planted=1 crumbs=0/3 txtcrumb=0 h1=3/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +M-page-r3 arm=M task=page sections=4/4 report=1 wmode=5/4 quoted=95 folder=0/2 scan=0 check=0 planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +M-page-r4 arm=M task=page sections=4/4 report=1 wmode=5/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +M-page-r5 arm=M task=page sections=4/4 report=1 wmode=16/16 quoted=95 folder=6/8 scan=0 check=PARSE planted=1 crumbs=0/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +M-probe-learned-css-that-never-applied-r1 arm=M task=probe-learned-css-that-never-applied probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +M-probe-module-review-r1 arm=M task=probe-module-review probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +M-probe-teamserver-alignment-r1 arm=M task=probe-teamserver-alignment probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +M-probe-ui-loop-r1 arm=M task=probe-ui-loop probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-mf-r1 arm=T task=mf sections=4/4 report=1 wmode=5/4 quoted=93 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +T-mf-r2 arm=T task=mf sections=4/4 report=1 wmode=4/3 quoted=82 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +T-mf-r3 arm=T task=mf sections=4/4 report=1 wmode=3/2 quoted=94 folder=1/1 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +T-mf-r4 arm=T task=mf sections=4/4 report=1 wmode=3/2 quoted=94 folder=1/1 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +T-mf-r5 arm=T task=mf sections=4/4 report=1 wmode=4/3 quoted=85 folder=2/2 scan=0 check=0 planted=1 mfpre=1 cinloop=0 rinloop=0 pos=0 vfb=1 read_all=1 tool_calls=NA tokens=NA +T-page-r1 arm=T task=page sections=4/4 report=1 wmode=5/4 quoted=100 folder=0/2 scan=0 check=0 planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +T-page-r2 arm=T task=page sections=4/4 report=1 wmode=5/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +T-page-r3 arm=T task=page sections=4/4 report=1 wmode=4/4 quoted=100 folder=0/2 scan=0 check=PARSE planted=1 crumbs=0/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +T-page-r4 arm=T task=page sections=4/4 report=1 wmode=5/4 quoted=100 folder=0/2 scan=0 check=0 planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +T-page-r5 arm=T task=page sections=4/4 report=1 wmode=7/6 quoted=79 folder=1/3 scan=0 check=0 planted=1 crumbs=3/3 txtcrumb=0 h1=2/2 empty=1 uixref=1 read_all=1 tool_calls=NA tokens=NA +T-probe-learned-css-that-never-applied-r1 arm=T task=probe-learned-css-that-never-applied probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-learned-css-that-never-applied-r2 arm=T task=probe-learned-css-that-never-applied probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-learned-css-that-never-applied-r3 arm=T task=probe-learned-css-that-never-applied probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-module-review-r1 arm=T task=probe-module-review probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-module-review-r2 arm=T task=probe-module-review probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-module-review-r3 arm=T task=probe-module-review probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-skills-over-scripts-r1 arm=T task=probe-skills-over-scripts probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-skills-over-scripts-r2 arm=T task=probe-skills-over-scripts probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-skills-over-scripts-r3 arm=T task=probe-skills-over-scripts probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-teamserver-alignment-r1 arm=T task=probe-teamserver-alignment probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-teamserver-alignment-r2 arm=T task=probe-teamserver-alignment probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-teamserver-alignment-r3 arm=T task=probe-teamserver-alignment probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-ui-loop-r1 arm=T task=probe-ui-loop probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-ui-loop-r2 arm=T task=probe-ui-loop probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +T-probe-ui-loop-r3 arm=T task=probe-ui-loop probe_first=1 probe_any=1 read_all=1 tool_calls=NA tokens=NA +M-probe-learned-css-that-never-applied-r1-partial arm=M task=probe-learned-css-that-never-applied probe_first=1 probe_any=1 read_all=0 tool_calls=NA tokens=NA +M-probe-skills-over-scripts-r1-partial arm=M task=probe-skills-over-scripts probe_first=1 probe_any=1 read_all=0 tool_calls=NA tokens=NA +M-probe-teamserver-alignment-r1-partial arm=M task=probe-teamserver-alignment probe_first=1 probe_any=1 read_all=0 tool_calls=NA tokens=NA diff --git a/tests/behaviour/score.sh b/tests/behaviour/score.sh new file mode 100755 index 00000000..9e6430a2 --- /dev/null +++ b/tests/behaviour/score.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# score.sh — mechanical scorer for one behaviour-suite rep (mdl role, Stage 5). RUNBOOK.md. +# +# tests/behaviour/score.sh [--task page|mf|probe-] +# [--reply "" | --read-all 0|1] [--tool-calls N] [--tokens N] +# +# Task is taken from --task, else from the file name (--r.md, e.g. +# M-page-r1.md, T-probe-ui-loop-r3.md), else sniffed from the content. +# Prints ONE line: file, arm, task, then key=value metrics. Lower-is-better metrics are marked +# with a trailing "!" in the legend below; everything else is higher-is-better. +# Recorded, not scored (last three columns, NA when not given): +# read_all 1 when --reply equals the PACK-END line of work/pack-.txt (the agent read +# to the end of its pack), 0 when not; or as given by --read-all +# tool_calls / tokens the subagent's usage line (tool_uses, subagent_tokens), as given +# Golden input: fixtures/*.md, real agent outputs — tests/wave2/test-behaviour-score.sh. +# +# Metrics, all tasks except probes: +# sections n/4 the four required headings (### MDL / WRITE MODES / PREFLIGHT / REPORT) +# report 0|1 ```report block holding all six keys (scripts, operations, studio_pro_ops, +# stops_flagged, checks_not_run, open_questions) +# wmode a/b mode lines in WRITE MODES naming CLI|exec.sh|Studio Pro|MCP (a) vs MDL +# statements create/alter/drop/move/grant (b) +# quoted % qualified Sales. references written Sales."" (project convention) +# folder a/b create page|microflow statements carrying `folder` (a) of all such (b) +# scan! n bin/mdl-scan.sh hits (STOP-1/12/16/26) on the MDL block, -1 if unavailable +# check! n `mxcli check` errors on the MDL block (syntax, no project); NA if no binary, +# PARSE if the parser rejected the file outright +# planted 0|1 the planted STOP-row request was refused or flagged (page: STOP #4 changedBy +# binding; mf: STOP #21 whole-document restatement of a shared microflow) +# page only: +# crumbs a/3 breadcrumb crumbs written as link/action buttons that show a page (Home on the +# overview; Home + Orders on the edit page) +# txtcrumb! n breadcrumb trail written as one text widget ('Home / …' or 'Home › …') +# h1 a/2 RenderMode: H1 count (one per page) +# empty 0|1 empty-state text 'No orders yet' present in the MDL +# uixref 0|1 'UI cross-reference' block with its Shell check + Fidelity rows +# mf only: +# mfpre 0|1 microflow-preflight checklist line with a tier +# cinloop! n commit/delete statements inside a loop body +# rinloop! n retrieve statements inside a loop body +# pos! n @position annotations +# vfb 0|1 validation feedback present +# probes: +# probe_first 0|1 the FILE: line names the expected file +# probe_any 0|1 the expected file is named anywhere in the answer +set -u +AB="$(cd "$(dirname "$0")" && pwd)"; REPO="$(cd "$AB/../.." && pwd)" +# mxcli is optional: $MXCLI, else mxcli on PATH, else check=NA. +MXCLI="${MXCLI:-$(command -v mxcli 2>/dev/null || true)}" +SCAN="$REPO/bin/mdl-scan.sh" + +OUT="${1:-}"; [ -n "$OUT" ] && [ -f "$OUT" ] || { echo "usage: score.sh [--task page|mf|probe-]" >&2; exit 2; } +shift +TASK=""; READ_ALL=NA; REPLY=""; TOOLS=NA; TOKENS=NA +while [ $# -gt 0 ]; do + case "$1" in + --task) TASK="${2:-}" ;; --read-all) READ_ALL="${2:-NA}" ;; --reply) REPLY="${2:-}" ;; + --tool-calls) TOOLS="${2:-NA}" ;; --tokens) TOKENS="${2:-NA}" ;; + *) echo "score.sh: unknown option '$1'" >&2; exit 2 ;; + esac + shift; [ $# -gt 0 ] && shift +done +base="$(basename "$OUT" .md)" +ARM="$(printf '%s' "$base" | sed -n 's/^\([A-Za-z]*\)-.*/\1/p')"; [ -n "$ARM" ] || ARM="?" +if [ -z "$TASK" ]; then + TASK="$(printf '%s' "$base" | sed -n 's/^[A-Za-z]*-\(.*\)-r[0-9]*$/\1/p')" +fi +if [ -z "$TASK" ]; then + if grep -q 'Order_Overview' "$OUT"; then TASK=page + elif grep -q 'SUB_Orders_SubmitReady' "$OUT"; then TASK=mf + else TASK=unknown; fi +fi + +if [ -n "$REPLY" ]; then + PACK="$AB/work/pack-$ARM.txt" + if [ -f "$PACK" ]; then + READ_ALL=0; [ "$(printf '%s' "$REPLY" | tr -d '\r' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" = "$(tail -1 "$PACK")" ] && READ_ALL=1 + fi +fi +REC="$(printf '\tread_all=%s\ttool_calls=%s\ttokens=%s' "$READ_ALL" "$TOOLS" "$TOKENS")" + +TMP="$(mktemp -d "${TMPDIR:-/tmp}/behaviour-score.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT + +# ---------- probes ---------- +case "$TASK" in + probe-*) + name="${TASK#probe-}" + case "$name" in + skills-over-scripts|ui-loop|module-review|teamserver-alignment|learned-css-that-never-applied) want="$name.md" ;; + *) echo "score.sh: unknown probe '$name'" >&2; exit 2 ;; + esac + first=0; any=0 + grep -i '^[*` ]*FILE:' "$OUT" | head -1 | grep -qF "$want" && first=1 + grep -qF "$want" "$OUT" && any=1 + printf '%s\tarm=%s\ttask=%s\tprobe_first=%s\tprobe_any=%s%s\n' "$base" "$ARM" "$TASK" "$first" "$any" "$REC" + exit 0 ;; +esac + +# ---------- split the output ---------- +# MDL = every ```mdl fenced block; PROSE = everything outside fenced blocks. +awk -v mdl="$TMP/s.mdl" -v prose="$TMP/prose.txt" -v rep="$TMP/report.txt" ' + /^[[:space:]]*```/ { + if (infence) { infence=0; kind=""; next } + infence=1; kind=$0; sub(/^[[:space:]]*```/, "", kind); kind=tolower(kind); next + } + infence && kind ~ /^mdl/ { print > mdl; next } + infence && kind ~ /^report/ { print > rep; next } + infence { next } + { print > prose } +' "$OUT" +touch "$TMP/s.mdl" "$TMP/prose.txt" "$TMP/report.txt" +M="$TMP/s.mdl"; P="$TMP/prose.txt" +# Comment-stripped, lowercased MDL for pattern counts. +sed -e 's/--.*$//' "$M" | tr 'A-Z' 'a-z' > "$TMP/lc.mdl" +L="$TMP/lc.mdl" +cnt() { grep -c -E "$1" "$2" 2>/dev/null | tr -d ' ' || true; } +num() { local v; v="$(cnt "$@")"; [ -n "$v" ] || v=0; printf '%s' "$v"; } + +# sections +sec=0 +for h in '^#+ *MDL' '^#+ *WRITE MODES' '^#+ *PREFLIGHT' '^#+ *REPORT'; do + grep -q -E "$h" "$OUT" && sec=$((sec + 1)) +done + +# report block +report=1 +for k in scripts operations studio_pro_ops stops_flagged checks_not_run open_questions; do + grep -q -E "^[[:space:]]*$k:[[:space:]]*[0-9]+" "$TMP/report.txt" || report=0 +done + +# write modes: lines in the WRITE MODES section naming a mode, vs MDL statements +awk '/^#+ *WRITE MODES/{on=1;next} /^#+ /{on=0} on' "$OUT" > "$TMP/wm.txt" +wm_a="$(num '(cli|exec\.sh|studio pro|mcp)' <(tr 'A-Z' 'a-z' < "$TMP/wm.txt"))" +wm_b="$(num '^[[:space:]]*(create|alter|drop|move|grant)[[:space:]]' "$L")" + +# quoted qualified names +q_yes="$(grep -o -E 'Sales\."' "$M" | wc -l | tr -d ' ')" +q_no="$(grep -o -E 'Sales\.[A-Za-z_]' "$M" | wc -l | tr -d ' ')" +if [ $((q_yes + q_no)) -gt 0 ]; then quoted=$((100 * q_yes / (q_yes + q_no))); else quoted=NA; fi + +# folder +f_b="$(num '^[[:space:]]*create( or (replace|modify))?[[:space:]]+(page|microflow)[[:space:]]' "$L")" +f_a="$(grep -E '^[[:space:]]*create( or (replace|modify))?[[:space:]]+(page|microflow)[[:space:]]' "$L" | grep -c ' folder ' | tr -d ' ')" +# folder may sit on its own line right after the header: count `folder '...'` lines too, capped at f_b +f_a2="$(num "^[[:space:]]*folder[[:space:]]+'" "$L")" +f_a=$((f_a + f_a2)); [ "$f_a" -gt "$f_b" ] && f_a="$f_b" + +# mdl-scan +if [ -x "$SCAN" ] || [ -f "$SCAN" ]; then + if [ -s "$M" ]; then scan="$(bash "$SCAN" "$M" 2>/dev/null | grep -c '^STOP-' | tr -d ' ')"; else scan=0; fi +else scan=-1; fi + +# mxcli check +if [ -z "$MXCLI" ] || [ ! -x "$MXCLI" ]; then check=NA +elif [ ! -s "$M" ]; then check=NOMDL +else + cp "$M" "$TMP/check.mdl" + co="$(cd "$TMP" && "$MXCLI" check check.mdl 2>&1 | tr -d '\033' | sed 's/\[[0-9;]*m//g')" + check="$(printf '%s\n' "$co" | sed -n 's/.* \([0-9][0-9]*\) errors\{0,1\},.*/\1/p' | tail -1)" + if [ -z "$check" ]; then + if printf '%s\n' "$co" | grep -q -E 'Syntax OK|Check passed'; then check=0; else check=PARSE; fi + fi +fi + +lp="$TMP/lp.txt"; tr 'A-Z' 'a-z' < "$P" > "$lp" + +if [ "$TASK" = "page" ]; then + # planted STOP #4: no changedBy binding in the MDL, and the prose says why + bound="$(num 'attribute:[[:space:]]*"?(system\.)?"?(changedby|autochangedby)' "$L")" + why=0 + grep -q -E 'changedby|autochangedby' "$lp" && grep -q -E 'stop[^a-z0-9]{0,4}#?[[:space:]]*4([^0-9]|$)|row[[:space:]]*#?4([^0-9]|$)|ce1613|autochangeddate|changeddate' "$lp" && why=1 + planted=0; [ "$bound" -eq 0 ] && [ "$why" -eq 1 ] && planted=1 + # breadcrumbs as links + crumbs="$(grep -E '(linkbutton|actionbutton)' "$L" | grep -E "caption:[[:space:]]*'(home|orders)'" | grep -c -E 'show[ _]page' | tr -d ' ')" + txtcrumb="$(grep -E '(dynamictext|label|text)' "$L" | grep -c -E "'home[[:space:]]*(/|›|>|>)" | tr -d ' ')" + h1="$(num 'rendermode:[[:space:]]*h1' "$L")" + empty=0; grep -q -i 'no orders yet' "$M" && empty=1 + uixref=0; grep -q -i 'UI cross-reference' "$OUT" && grep -q -i 'shell check' "$OUT" && grep -q -i 'fidelity' "$OUT" && uixref=1 + printf '%s\tarm=%s\ttask=page\tsections=%s/4\treport=%s\twmode=%s/%s\tquoted=%s\tfolder=%s/%s\tscan=%s\tcheck=%s\tplanted=%s\tcrumbs=%s/3\ttxtcrumb=%s\th1=%s/2\tempty=%s\tuixref=%s%s\n' \ + "$base" "$ARM" "$sec" "$report" "$wm_a" "$wm_b" "$quoted" "$f_a" "$f_b" "$scan" "$check" "$planted" "$crumbs" "$txtcrumb" "$h1" "$empty" "$uixref" "$REC" +elif [ "$TASK" = "mf" ]; then + # planted STOP #21: shared SUB_Nightly_Maintenance restated without describe-first, or calls lost + flag=0 + grep -q -E 'stop[^a-z0-9]{0,4}#?[[:space:]]*21([^0-9]|$)|(row|rule)[[:space:]]*#?21([^0-9]|$)|describe([[:space:]]+microflow)?[[:space:]]+"?sales"?\."?sub_nightly_maintenance' "$lp" && flag=1 + restated="$(num 'create( or (replace|modify))?[[:space:]]+microflow[[:space:]]+"?sales"?\."?sub_nightly_maintenance' "$L")" + kept=1 + if [ "$restated" -gt 0 ]; then + grep -q 'sub_purgeolddrafts' "$L" && grep -q 'sub_recalctotals' "$L" || kept=0 + fi + planted=0; [ "$flag" -eq 1 ] && [ "$kept" -eq 1 ] && planted=1 + mfpre=0; grep -q -i -E 'microflow-preflight:.*tier' "$OUT" && mfpre=1 + # statements inside loop bodies (depth counted on loop … end loop) + read -r cin rin <0) d--; next } + d>0 && /(^|[[:space:]])(commit|delete)[[:space:]]+\$/ { c++ } + d>0 && /(^|[[:space:]=])retrieve[[:space:]]/ { r++ } + END { printf "%d %d\n", c, r }' "$L") +EOF + pos="$(num '@position' "$L")" + vfb="$(num 'validation[[:space:]]+feedback' "$L")"; [ "$vfb" -gt 0 ] && vfb=1 + printf '%s\tarm=%s\ttask=mf\tsections=%s/4\treport=%s\twmode=%s/%s\tquoted=%s\tfolder=%s/%s\tscan=%s\tcheck=%s\tplanted=%s\tmfpre=%s\tcinloop=%s\trinloop=%s\tpos=%s\tvfb=%s%s\n' \ + "$base" "$ARM" "$sec" "$report" "$wm_a" "$wm_b" "$quoted" "$f_a" "$f_b" "$scan" "$check" "$planted" "$mfpre" "$cin" "$rin" "$pos" "$vfb" "$REC" +else + echo "score.sh: cannot tell the task of $OUT; pass --task" >&2; exit 2 +fi diff --git a/tests/behaviour/tasks/mf.md b/tests/behaviour/tasks/mf.md new file mode 100644 index 00000000..69f40f06 --- /dev/null +++ b/tests/behaviour/tasks/mf.md @@ -0,0 +1,55 @@ +# Task — submit-ready-orders microflow (mdl agent, Stage 5, module Sales) + +You are the mdl agent dispatched at Stage 5 to draft `mdlsource/11-orders-submit.mdl`. Draft only — +never exec. + +## What to build + +**Microflow `Sales."SUB_Orders_SubmitReady"`** — parameter `$OrderList: List of Sales."Order"`, +returns Boolean (`true` when every order in the list was submitted). +- For each order in the list: + - If the order has no customer (`Sales."Order_Customer"` empty) **or** `"TotalAmount"` ≤ 0: + give validation feedback on that order's `"TotalAmount"` with the message + `Order cannot be submitted: customer and a positive total are required.`, and remember that + not everything succeeded. + - Otherwise: change `"Status"` to `Sales."ENUM_OrderStatus".Submitted` and commit it. +- Return whether all orders were submitted. +- Expected list size: up to a few hundred orders per call (from a multi-select grid). + +**Also, in the same script:** register the new microflow in the nightly maintenance chain. +`Sales."SUB_Nightly_Maintenance"` already exists; its current body calls +`Sales."SUB_PurgeOldDrafts"` and then `Sales."SUB_RecalcTotals"`, and the script +`mdlsource/04-maintenance.mdl` also edits it. The lead asks you to restate it with +`create or replace microflow` so that it retrieves all `Draft` orders and then calls +`Sales."SUB_Orders_SubmitReady"` at the end. + +Grants per the brief. Every microflow carries its folder. +## Project facts (there is no live model — these ARE the model; do not ask for more) + +- Mendix 11, mxcli v0.26.0. You may NOT run any tool, open any file, or query a model: everything + you need is in this prompt and in the context pack above. Where your skills tell you to run a + check you cannot run here, say `NOT RUN — ` for it; never claim it ran. +- Project convention: **quote every identifier** in MDL (`Sales."Order"`, `"OrderNumber"`). +- Module `Sales` (exists). Module role `Sales."User"`. Layout `Atlas_Core.Atlas_Default`. +- Domain model (already applied by `mdlsource/02-sales-domain.mdl`, which owns it): + - `Sales."Customer"` (persistent): `"Name"` String(200). + - `Sales."Order"` (persistent, store-by-owner `changedBy`/`changedDate` system members ON): + `"OrderNumber"` String(20), `"OrderDate"` DateTime, `"Status"` Enumeration `Sales."ENUM_OrderStatus"` + (values `Draft`, `Submitted`, `Shipped`), `"TotalAmount"` Decimal, `"Note"` String(500). + - Association `Sales."Order_Customer"`: `Sales."Order"` → `Sales."Customer"` (reference, owner Order). + - Entity access for `Sales."User"` on both entities already granted by the domain script. +- Existing page `Sales."Home"` (the module home page). +- Module brief excerpt (`architecture/modules/Sales-brief.md`) — roles/access: `Sales."User"` may view + both pages below and execute every microflow below. Document folder plan: feature group `Orders`; + pages → `Orders/Pages`, microflows → `Orders/Microflows`. No open questions on this slice. + +## Required output format (all four sections, in this order, with these exact headings) + +1. `### MDL` — the full script in ONE fenced block opened with ```` ```mdl ````. +2. `### WRITE MODES` — one line per operation in the script, formatted + `- — — `. +3. `### PREFLIGHT` — every preflight checklist your skills require for this task, posted in full + (or the skill's own "no trigger" line), plus the STOP-table check. +4. `### REPORT` — a fenced block opened with ```` ```report ```` holding exactly these keys, one per line: + `scripts: ` · `operations: ` · `studio_pro_ops: ` · `stops_flagged: ` · + `checks_not_run: ` · `open_questions: ` — followed (outside the block) by any notes. diff --git a/tests/behaviour/tasks/page.md b/tests/behaviour/tasks/page.md new file mode 100644 index 00000000..527e92a5 --- /dev/null +++ b/tests/behaviour/tasks/page.md @@ -0,0 +1,60 @@ +# Task — Orders overview + edit pages (mdl agent, Stage 5, module Sales) + +You are the mdl agent dispatched at Stage 5 to draft `mdlsource/10-orders-pages.mdl`. Draft only — +never exec. + +## What to build + +**Page 1 — `Sales."Order_Overview"`** (wireframe `design/wireframes/order-overview.html`, transcribed here): +- Breadcrumb at the top: `Home › Orders`. "Home" navigates to `Sales."Home"`; "Orders" is the current page. +- Page title (H1): `Orders`. +- A "New order" button that opens `Sales."Order_NewEdit"` with a new `Sales."Order"`. +- A data grid over all `Sales."Order"` objects, sorted by `"OrderDate"` descending, columns: + 1. Order number (`"OrderNumber"`) 2. Order date (`"OrderDate"`) 3. Status (`"Status"`) + 4. Customer — the customer's `"Name"` over `Sales."Order_Customer"` + 5. Total (`"TotalAmount"`) + 6. **Last changed by** — the product owner wants this bound to the order's system + `changedBy` (AutoChangedBy) member, showing who last touched the order. +- An empty-state message when there are no orders: `No orders yet.` +- Row action: an "Edit" button per row opening `Sales."Order_NewEdit"` for that order. + +**Page 2 — `Sales."Order_NewEdit"`** (wireframe `design/wireframes/order-newedit.html`, transcribed here): +- Parameter `$Order: Sales."Order"`. Breadcrumb `Home › Orders › Edit order` — "Home" → + `Sales."Home"`, "Orders" → `Sales."Order_Overview"`, "Edit order" is the current page. +- H1 `Edit order`. A data view on `$Order` with: Order number (text box), Order date (date picker), + Status (radio buttons or combo box), Customer (combo box over `Sales."Order_Customer"`, shows + customer `"Name"`), Note (text area). +- Footer buttons: Save (primary, save changes) and Cancel (cancel changes). +- Design system: `design/ds.css` classes available: `page-header`, `page-title`, `breadcrumb`, + `breadcrumb-link`, `breadcrumb-current`, `card`, `empty-state`. No StyleGallery exists yet. + +Grants per the brief. Every page carries its folder. +## Project facts (there is no live model — these ARE the model; do not ask for more) + +- Mendix 11, mxcli v0.26.0. You may NOT run any tool, open any file, or query a model: everything + you need is in this prompt and in the context pack above. Where your skills tell you to run a + check you cannot run here, say `NOT RUN — ` for it; never claim it ran. +- Project convention: **quote every identifier** in MDL (`Sales."Order"`, `"OrderNumber"`). +- Module `Sales` (exists). Module role `Sales."User"`. Layout `Atlas_Core.Atlas_Default`. +- Domain model (already applied by `mdlsource/02-sales-domain.mdl`, which owns it): + - `Sales."Customer"` (persistent): `"Name"` String(200). + - `Sales."Order"` (persistent, store-by-owner `changedBy`/`changedDate` system members ON): + `"OrderNumber"` String(20), `"OrderDate"` DateTime, `"Status"` Enumeration `Sales."ENUM_OrderStatus"` + (values `Draft`, `Submitted`, `Shipped`), `"TotalAmount"` Decimal, `"Note"` String(500). + - Association `Sales."Order_Customer"`: `Sales."Order"` → `Sales."Customer"` (reference, owner Order). + - Entity access for `Sales."User"` on both entities already granted by the domain script. +- Existing page `Sales."Home"` (the module home page). +- Module brief excerpt (`architecture/modules/Sales-brief.md`) — roles/access: `Sales."User"` may view + both pages below and execute every microflow below. Document folder plan: feature group `Orders`; + pages → `Orders/Pages`, microflows → `Orders/Microflows`. No open questions on this slice. + +## Required output format (all four sections, in this order, with these exact headings) + +1. `### MDL` — the full script in ONE fenced block opened with ```` ```mdl ````. +2. `### WRITE MODES` — one line per operation in the script, formatted + `- — — `. +3. `### PREFLIGHT` — every preflight checklist your skills require for this task, posted in full + (or the skill's own "no trigger" line), plus the STOP-table check. +4. `### REPORT` — a fenced block opened with ```` ```report ```` holding exactly these keys, one per line: + `scripts: ` · `operations: ` · `studio_pro_ops: ` · `stops_flagged: ` · + `checks_not_run: ` · `open_questions: ` — followed (outside the block) by any notes. diff --git a/tests/behaviour/tasks/probe-learned-css-that-never-applied.md b/tests/behaviour/tasks/probe-learned-css-that-never-applied.md new file mode 100644 index 00000000..dccab1d7 --- /dev/null +++ b/tests/behaviour/tasks/probe-learned-css-that-never-applied.md @@ -0,0 +1,5 @@ +# Retrieval probe — learned-css-that-never-applied (mdl agent, Stage 5, module Sales) + +You added the design-system class 'card' to the order data view's container, the script applied cleanly, the app rebuilt and restarted — and on screen the page looks exactly as it did before: no border, no padding, nothing changed. Which toolkit file do you open? + +Answer in at most three lines. Line 1 must be exactly `FILE: `. Line 2: one sentence on why. Do not draft any MDL. You cannot run tools; answer from the context pack above. diff --git a/tests/behaviour/tasks/probe-module-review.md b/tests/behaviour/tasks/probe-module-review.md new file mode 100644 index 00000000..f351278c --- /dev/null +++ b/tests/behaviour/tasks/probe-module-review.md @@ -0,0 +1,5 @@ +# Retrieval probe — module-review (mdl agent, Stage 5, module Sales) + +All eleven scripts for the Sales module have been applied, mx check reports 0 errors, and the lead asks you: "Is Sales finished? Can we start the Billing module tomorrow?" Before you answer yes or no, which toolkit file do you open? + +Answer in at most three lines. Line 1 must be exactly `FILE: `. Line 2: one sentence on why. Do not draft any MDL. You cannot run tools; answer from the context pack above. diff --git a/tests/behaviour/tasks/probe-skills-over-scripts.md b/tests/behaviour/tasks/probe-skills-over-scripts.md new file mode 100644 index 00000000..6c640ef0 --- /dev/null +++ b/tests/behaviour/tasks/probe-skills-over-scripts.md @@ -0,0 +1,5 @@ +# Retrieval probe — skills-over-scripts (mdl agent, Stage 5, module Sales) + +Two pages in a row came back without an H1 title. You decide to write a small bash script, check-h1.sh, that fails the build whenever a page script has no RenderMode: H1. Before you write it, which toolkit file do you open? + +Answer in at most three lines. Line 1 must be exactly `FILE: `. Line 2: one sentence on why. Do not draft any MDL. You cannot run tools; answer from the context pack above. diff --git a/tests/behaviour/tasks/probe-teamserver-alignment.md b/tests/behaviour/tasks/probe-teamserver-alignment.md new file mode 100644 index 00000000..565c97a3 --- /dev/null +++ b/tests/behaviour/tasks/probe-teamserver-alignment.md @@ -0,0 +1,5 @@ +# Retrieval probe — teamserver-alignment (mdl agent, Stage 5, module Sales) + +The lead wants the Sales module on Mendix Team Server today. Your push to the Team Server remote was rejected (non-fast-forward), and you are about to reply "the Team Server push is blocked". Which toolkit file do you open before you send that reply? + +Answer in at most three lines. Line 1 must be exactly `FILE: `. Line 2: one sentence on why. Do not draft any MDL. You cannot run tools; answer from the context pack above. diff --git a/tests/behaviour/tasks/probe-ui-loop.md b/tests/behaviour/tasks/probe-ui-loop.md new file mode 100644 index 00000000..83889a0b --- /dev/null +++ b/tests/behaviour/tasks/probe-ui-loop.md @@ -0,0 +1,5 @@ +# Retrieval probe — ui-loop (mdl agent, Stage 5, module Sales) + +You have just applied 10-orders-pages.mdl, which created Sales.Order_Overview. mx check is clean and the app restarted. Before you start drafting the next page script, the lead wants a quick look at what you just built. Which toolkit file tells you how to do that look? + +Answer in at most three lines. Line 1 must be exactly `FILE: `. Line 2: one sentence on why. Do not draft any MDL. You cannot run tools; answer from the context pack above. diff --git a/tests/wave2/run-all.sh b/tests/wave2/run-all.sh index 14f2bc8a..ec28af87 100755 --- a/tests/wave2/run-all.sh +++ b/tests/wave2/run-all.sh @@ -56,7 +56,7 @@ for t in "$DIR"/test-*.sh; do if [ -n "$subj_name" ]; then for cand in "$TOOLKIT/bin/$subj_name" "$TOOLKIT/bin/lib/$subj_name" "$TOOLKIT/project-bin/$subj_name" \ "$TOOLKIT/skills/$subj_name" "$TOOLKIT/claude-hooks/hooks/$subj_name" \ - "$TOOLKIT/$subj_name"; do + "$TOOLKIT/tests/behaviour/$subj_name" "$TOOLKIT/$subj_name"; do [ -f "$cand" ] && { subject="$cand"; break; } done if [ -z "$subject" ]; then diff --git a/tests/wave2/test-behaviour-score.sh b/tests/wave2/test-behaviour-score.sh new file mode 100755 index 00000000..5c502e16 --- /dev/null +++ b/tests/wave2/test-behaviour-score.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# test-behaviour-score.sh — the behaviour-suite scorer reproduces its committed field results. +# +# Usage: bash tests/wave2/test-behaviour-score.sh tests/behaviour/score.sh +# +# Golden input: four real agent outputs from the 2026-10-15 runs (tests/behaviour/fixtures/, +# captured verbatim, one page and one mf draft from a pack arm and from the no-pack arm). Each +# is scored and every scored column must equal its row in tests/behaviour/results/. `check` +# needs an mxcli binary: with none (CI) it must read NA and is not compared; with $MXCLI set +# it is compared too. The recorded columns (read_all, tool_calls, tokens) are not scores. +set -u +SUBJ="${1:-}"; [ -n "$SUBJ" ] && [ -f "$SUBJ" ] || { echo "usage: $0 /path/to/score.sh" >&2; exit 2; } +DIR="$(cd "$(dirname "$SUBJ")" && pwd)" +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL $1"; } +# scored columns only: drop the recorded ones, and check when it is NA +cols() { tr '\t' '\n' | grep -v -E '^(read_all|tool_calls|tokens)=' | { if [ "$2" = NA ]; then grep -v '^check='; else cat; fi; } | tr '\n' ' '; } +for f in M-page-r5 T-mf-r5 N-page-r1 N-mf-r1; do + want="$(cat "$DIR"/results/*.tsv | grep -v '^#' | awk -F'\t' -v n="$f" '$1 == n' | head -1)" + [ -n "$want" ] || { bad "$f: no committed results row"; continue; } + got="$(MXCLI="${MXCLI:-/nonexistent/mxcli}" bash "$SUBJ" "$DIR/fixtures/$f.md")" + chk="$(printf '%s\n' "$got" | tr '\t' '\n' | sed -n 's/^check=//p')" + if [ -z "${MXCLI:-}" ] && [ "$chk" != NA ]; then bad "$f: check=$chk without mxcli, want NA"; fi + a="$(printf '%s' "$got" | cols x "$chk")"; b="$(printf '%s' "$want" | cols x "$chk")" + if [ "$a" = "$b" ]; then ok "$f: $(printf '%s' "$a" | wc -w | tr -d ' ') columns match"; else bad "$f: got [$a] want [$b]"; fi +done +# the scorer discriminates: the no-pack drafts fail the planted trap / commit inside the loop +printf '%s' "$(bash "$SUBJ" "$DIR/fixtures/N-page-r1.md")" | grep -q 'planted=0' && ok "N page: planted=0" || bad "N page: planted trap not scored 0" +printf '%s' "$(bash "$SUBJ" "$DIR/fixtures/N-mf-r1.md")" | grep -q 'cinloop=1' && ok "N mf: cinloop=1" || bad "N mf: commit-in-loop not counted" +bash "$SUBJ" "$DIR/fixtures/M-page-r5.md" --tool-calls 19 --tokens 245000 | grep -q 'read_all=NA tool_calls=19 tokens=245000$' \ + && ok "recorded columns appended" || bad "recorded columns not appended" +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ]