diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 4f9ccdd9..26a95a21 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -100,3 +100,18 @@ jobs: run: bash tests/lint-rules/test-batch3-rules.sh - name: Lint-rule unit suite, batch 4 (tests/lint-rules/test-batch4-rules.sh) run: bash tests/lint-rules/test-batch4-rules.sh + + # ADVISORY, never a gate (added with tests/advisory/authoring-audit.py): prints near-duplicate + # rules across skills/agents/CLAUDE.md and routing top-3 hits for the labelled phrasings + # (CLAUDE.md rule 8). Its own job, so it reports even when a gate above is red, and it cannot + # turn the PR red: continue-on-error on job and step, `|| true` on both runs. Stdlib Python, ~15 s. + authoring-audit: + runs-on: ubuntu-latest + continue-on-error: true + steps: + - uses: actions/checkout@v4 + - name: Authoring audit, advisory (routing, dupes) + continue-on-error: true + run: | + python3 tests/advisory/authoring-audit.py routing || true + python3 tests/advisory/authoring-audit.py dupes --top 40 || true diff --git a/CHANGELOG.md b/CHANGELOG.md index 994f1202..13fcaf6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- process(tests/advisory): **maintainer-side, advisory authoring audit: `tests/advisory/authoring-audit.py` reports near-duplicate rules and routing misses; it never ships and never gates.** `dupes` chunks `skills/`, `agents/`, `CLAUDE.md` and README "Baseline routing" into rule-sized pieces and ranks cross-file pairs by sparse TF-IDF cosine on 1–2-grams (ROUTING blocks, same-file and agent-to-agent pairs skipped, max 3 per file pair, `dupes-allow.tsv` seeded with the declared architect-agent ↔ workflow-structure-rules mirror and the pointer/tombstone/boilerplate pairs). `routing` scores BM25 over each `skill-routing.tsv` trigger against 55 labelled phrasings (`routing-phrasings.tsv`) and lists every top-3 miss; `--strict` exits 1 only for a `must_hit` row, and none is marked yet. Python 3 stdlib only. New non-blocking CI job `authoring-audit`; fixture `tests/wave2/test-authoring-audit.sh` (5 checks, floor top-3 ≥ 44/55); `run-all.sh` resolves `.py` subjects and `tests/advisory/`; CLAUDE.md rule 8 points at it. Field evidence, master 484e7ff: TF-IDF put 19 true duplicates in its top 40 (82.5% not false positives over 69 hand-judged pairs; 5-shingle Jaccard was worse); BM25 on trigger text hit top-3 46/55 (top-1 36/55), matching or beating an LSA stand-in; `dupes` 16 s over 7,031 chunks, `routing` < 0.1 s. — authoring audit, maintainer request - new(tests/behaviour): **a standing behaviour suite for agent-read text: arms M (base ref), T (candidate ref, or the base minus one skill for an ablation) and N (no pack), two drafting tasks with planted STOP-row requests and five retrieval probes, scored mechanically.** `tests/behaviour/build-packs.sh` builds a pack from any ref in a temporary worktree; `score.sh` adds `read_all` (the agent quoted the pack's `PACK-END` line back), `tool_calls` and `tokens`, and treats `mxcli check` as optional (NA without a binary). `RUNBOOK.md` has the triggers (every PR that changes baseline-routed text or an agent stub; N and ablations at each release cut), 5 reps x 2 tasks plus 5 probes x 3 reps, the rule (T no worse than M on every scored metric, probes T >= M - 1), and partial reads counted and re-run, never dropped. `docs/RELEASE-PROCESS.md` §4a names it as the way to run the Behaviour A/B. Field runs committed under `tests/behaviour/results/`: the 2026-10-15 stub-diet A/B (#369/#378/#380, 42 reps, 3 partial reads on the 4,822-line pack) and the first no-pack run (page trap refused 0/5 without a pack against 5/5 with either; mf passes check 1/5 against 5/5; every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'). Golden input: four real agent outputs, `tests/wave2/test-behaviour-score.sh`. — maintainer request (MendixMau) - process(pr-template, release-process): **every Direct-lane PR and every review fills a five-row Impact block: Process, Dependencies, Context cost, Behaviour, Runtime.** "none" or "NOT RUN: " is legal, a blank row is not. Context cost is pasted from `bin/render-routing.sh --check` on base and head, never estimated; a missing or estimated row is a review finding. The Behaviour A/B (`process/preflight-skill-baseline-2026-08-31.md`) is required for baseline-routed text and agent stubs, recommended for situational skills, NOT RUN with a reason for process docs review agents read, n/a for scripts only; Context cost may be "none" when no routed or always-loaded file is touched. `docs/RELEASE-PROCESS.md` §4a holds the rule and a bad/good example; `CONTRIBUTING.md`'s reviewer list points there. No CI check: `bin/check-pr-discipline.sh` reads the git diff, not the PR body. Field evidence: three PRs reviewed on 2026-10-15 with no process cost stated (#375 added a human step per re-baseline, #384 adds 1–11 s per exec), both found by chance. — maintainer request, MendixMau - learn(learned-detection-gaps, testing-shape, ui-loop): **a class on a snippet call never reaches the DOM; a 403 bounce and a widget-pinned session probe both read as sign-outs; app restarts matched a "signs me out" report by timing.** Detection-gaps row: `Class` on a `SNIPPETCALL` is green through check, exec, mxbuild, lint and DESCRIBE, and absent in the browser ([#336](https://github.com/MendixMau/mxcli-project-toolkit/issues/336)). Testing-shape §4 gains two false-red rows: `login.html?messageCode=403` with the session kept ([#338](https://github.com/MendixMau/mxcli-project-toolkit/issues/338)), and instruments pinned to one grid name or a typed menu list (1/7 and 7/34 false FAILs after a redesign, [#339](https://github.com/MendixMau/mxcli-project-toolkit/issues/339)). ui-loop: say so before a restart while a human uses the app ([#337](https://github.com/MendixMau/mxcli-project-toolkit/issues/337)) — three-app field run, 2026-10-14 diff --git a/CLAUDE.md b/CLAUDE.md index 425608d5..57e1a346 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -285,6 +285,8 @@ Create `skills/{topic}.md` with `# Title`, `**Applies to:** migration | any mxcl produce (rule 1 makes the row a trigger condition; this rule is the check that the trigger works). The coverage audit in `render-routing.sh --check` catches the unrouted case mechanically; wrong-trigger routing it cannot see. + `tests/advisory/authoring-audit.py routing` is that check (BM25 over trigger text vs the + labelled `routing-phrasings.tsv`); its `dupes` lists rules stated twice. Advisory, never a gate. (Rules 6–8 adapted 2026-08-31 from public skill-authoring guidance in obra/superpowers `writing-skills` and Matt Pocock's `writing-for-agents` — patterns only. Their fourth diff --git a/tests/advisory/authoring-audit.py b/tests/advisory/authoring-audit.py new file mode 100755 index 00000000..098a8d80 --- /dev/null +++ b/tests/advisory/authoring-audit.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +"""authoring-audit.py -- maintainer-side, ADVISORY authoring audit. Never ships to projects. + + dupes near-duplicate rules across skills/, agents/, CLAUDE.md and README "Baseline routing" + (sparse TF-IDF cosine on 1-2-grams). Each hit is a rule stated twice: pick the canonical + home and make the other a pointer, or allowlist it in dupes-allow.tsv with a reason. + routing BM25 over the trigger text (column 3) of bin/lib/skill-routing.tsv, scored against the + labelled phrasings in routing-phrasings.tsv -- CLAUDE.md "Adding new skills", rule 8. + +Python 3 standard library only. Exit 0 always, except `routing --strict` (a must_hit phrasing missed +its top 3). Calibration on master 484e7ff: dupes top 40 held 19 true duplicates (82.5% not false +positives, 69 hand-judged pairs); routing top-3 46/55. + +Usage: authoring-audit.py dupes [--root DIR] [--top N] + authoring-audit.py routing [--root DIR] [--phrasings FILE] [--strict] +""" +import argparse, fnmatch, glob, math, os, re, sys, time +from collections import Counter, defaultdict + +HERE = os.path.dirname(os.path.abspath(__file__)) +# sklearn's English list is not available without sklearn; this covers the high-df function words. +STOP = set("""a about above after again against all also am an and any are as at be because been +before being below between both but by can could did do does doing down during each else few for +from further had has have having he her here hers him his how i if in into is it its itself just +me more most my no nor not now of off on once only or other our ours out over own same she should +so some such than that the their them then there these they this those through to too under until +up very was we were what when where which while who whom why will with would you your yours""".split()) + +# ---------------------------------------------------------------- chunking +LIST_RE = re.compile(r'^\s*([-*+]|\d+[.)])\s+') +def chunks_of(lines, lo=0, hi=None): + """Rule-sized chunks: one per table row, one per bullet (with its indented continuation), and + paragraphs capped at ~120 words. Code fences, headings and ROUTING:BEGIN/END blocks skipped.""" + out, para, pstart = [], [], None + hi = len(lines) if hi is None else hi + i, incode, inroute = lo, False, False + def flush(): + nonlocal para, pstart + if para: out.append((pstart, ' '.join(para))) + para, pstart = [], None + while i < hi: + raw = lines[i].rstrip('\n'); s = raw.strip() + if 'ROUTING:BEGIN' in s: inroute = True + if 'ROUTING:END' in s: inroute = False; i += 1; continue + if inroute: i += 1; continue + if s.startswith('```'): flush(); incode = not incode; i += 1; continue + if incode: i += 1; continue + if not s or s.startswith('#'): flush(); i += 1; continue + if s.startswith('|'): + flush() + if not re.fullmatch(r'\|[\s:\-|]+\|?', s): out.append((i + 1, s)) + i += 1; continue + if LIST_RE.match(raw): + flush(); buf, start = [s], i + 1; i += 1 + while (i < hi and lines[i].strip() and lines[i].startswith(' ') + and not re.match(r'^\s*([-*+]|\d+[.)]|\||#|```)', lines[i])): + buf.append(lines[i].strip()); i += 1 + out.append((start, ' '.join(buf))); continue + if pstart is None: pstart = i + 1 + para.append(s.lstrip('> ').strip()); i += 1 + if len(' '.join(para).split()) > 120: flush() + flush() + return out + +def corpus(root): + files = sorted(glob.glob(os.path.join(root, 'skills', '**', '*.md'), recursive=True)) + files += sorted(glob.glob(os.path.join(root, 'agents', '*.md'))) + [os.path.join(root, 'CLAUDE.md')] + out = [] + def add(rel, lines, lo=0, hi=None): + for ln, t in chunks_of(lines, lo, hi): + if len(re.findall(r'\w+', t)) >= 10: out.append((rel, ln, t)) + for f in files: + if os.path.isfile(f): + with open(f, encoding='utf-8') as fh: add(os.path.relpath(f, root).replace(os.sep, '/'), fh.readlines()) + with open(os.path.join(root, 'README.md'), encoding='utf-8') as fh: L = fh.readlines() + st = next((i for i, l in enumerate(L) if l.startswith('#### Baseline routing')), None) + if st is not None: + en = next((i for i in range(st + 1, len(L)) if re.match(r'^#{2,4} ', L[i])), len(L)) + add('README.md', L, st + 1, en) + return out + +# ---------------------------------------------------------------- dupes +def tfidf(texts): + """Sparse, L2-normalised TF-IDF over 1-2-grams (sublinear tf, smoothed idf). Returns dicts.""" + grams = [] + for t in texts: + w = [x for x in re.findall(r'\b\w\w+\b', re.sub(r'[`*_|>]', ' ', t).lower()) if x not in STOP] + grams.append(Counter(w + [a + ' ' + b for a, b in zip(w, w[1:])])) + df = Counter(g for c in grams for g in c) + n = len(texts); vecs = [] + for c in grams: + v = {g: (1 + math.log(tf)) * (math.log((1 + n) / (1 + df[g])) + 1) for g, tf in c.items()} + norm = math.sqrt(sum(x * x for x in v.values())) or 1.0 + vecs.append({g: x / norm for g, x in v.items()}) + return vecs + +def load_allow(path): + rows = [] + if os.path.isfile(path): + for l in open(path, encoding='utf-8'): + if l.strip() and not l.startswith('#'): + p = l.rstrip('\n').split('\t') + if len(p) >= 3: rows.append((p[0], p[1], p[2].lower())) + return rows + +def allowed(allow, fa, fb, ta, tb): + for ga, gb, m in allow: + if ((fnmatch.fnmatch(fa, ga) and fnmatch.fnmatch(fb, gb)) or + (fnmatch.fnmatch(fa, gb) and fnmatch.fnmatch(fb, ga))): + if m == '*' or m in ta.lower() or m in tb.lower(): return True + return False + +def cmd_dupes(a): + t0 = time.time() + C = corpus(a.root); V = tfidf([t for _, _, t in C]) + isag = [f.startswith('agents/') for f, _, _ in C] + post = defaultdict(list) # inverted index -> sparse dot products + for i, v in enumerate(V): + for g, x in v.items(): post[g].append((i, x)) + cand = [] + for i, v in enumerate(V): + acc = defaultdict(float) + for g, x in v.items(): + for j, y in post[g]: + if j > i: acc[j] += x * y + best = sorted(((s, j) for j, s in acc.items() + if C[j][0] != C[i][0] and not (isag[i] and isag[j])), reverse=True)[:5] + cand += [(s, i, j) for s, j in best if s > 0] + cand.sort(key=lambda c: (-c[0], c[1], c[2])) + allow = load_allow(os.path.join(HERE, 'dupes-allow.tsv')) + per, shown, skipped = Counter(), 0, 0 + print(f'dupes: {len(C)} chunks from {len({f for f, _, _ in C})} files; top {a.top} cross-file pairs ' + f'(max 3 per file pair), allowlist {len(allow)} rows') + for s, i, j in cand: + (fa, la, ta), (fb, lb, tb) = C[i], C[j] + key = tuple(sorted((fa, fb))) + if allowed(allow, fa, fb, ta, tb): skipped += 1; continue # before the cap: never spends it + if per[key] >= 3: continue + per[key] += 1 + shown += 1 + print(f'{shown:3d} {s:.3f} {fa}:{la} <> {fb}:{lb}\n A: {ta[:110]}\n B: {tb[:110]}') + if shown >= a.top: break + print(f'dupes: {shown} shown, {skipped} allowlisted, {time.time() - t0:.1f}s') + return 0 + +# ---------------------------------------------------------------- routing +def stem(w): + for s in ('ing', 'ed', 'es', 's'): + if len(w) > 4 and w.endswith(s): return w[:-len(s)] + return w +QSTOP = set('a an the of to and or in on for is it be at by as this that with i my me we our you ' + 'your its are was can do if not no'.split()) +def tok(t): return [stem(w) for w in re.findall(r'[a-z0-9]+', t.lower()) if w not in QSTOP] + +class BM25: # Okapi, k1=1.5 b=0.75, idf floor eps*mean + def __init__(self, docs, k1=1.5, b=0.75, eps=0.25): + self.tf = [Counter(d) for d in docs]; self.dl = [len(d) for d in docs] + self.avg = sum(self.dl) / len(docs); self.k1, self.b = k1, b + df = Counter(w for d in docs for w in set(d)); n = len(docs) + self.idf = {w: math.log(n - c + 0.5) - math.log(c + 0.5) for w, c in df.items()} + floor = eps * sum(self.idf.values()) / len(self.idf) + self.idf = {w: (x if x >= 0 else floor) for w, x in self.idf.items()} + def scores(self, q): + out = [] + for tf, dl in zip(self.tf, self.dl): + s = 0.0 + for w in q: + f = tf.get(w, 0) + if f: s += self.idf[w] * f * (self.k1 + 1) / (f + self.k1 * (1 - self.b + self.b * dl / self.avg)) + out.append(s) + return out + +def cmd_routing(a): + t0 = time.time() + rows = [l.rstrip('\n').split('\t') for l in open(os.path.join(a.root, 'bin/lib/skill-routing.tsv'), encoding='utf-8') + if l.strip() and not l.startswith('#')] + paths = [r[1] for r in rows]; bm = BM25([tok(r[2]) for r in rows]) + qs = [l.rstrip('\n').split('\t') for l in open(a.phrasings, encoding='utf-8') if l.strip() and not l.startswith('#')] + top1 = top3 = 0; misses = []; strict_fail = [] + for q in qs: + phr, exp = q[0], q[1] + ok = {exp} | {x for x in (q[2] if len(q) > 2 else '').split(',') if x and x != '-'} + s = bm.scores(tok(phr)) + order = [paths[i] for i in sorted(range(len(s)), key=lambda i: (-s[i], i))[:3]] + rank = min([order.index(p) + 1 for p in ok if p in order], default=99) + top1 += rank == 1; top3 += rank <= 3 + if rank > 3: + must = len(q) > 3 and q[3].strip() == 'must_hit' + misses.append((phr, exp, order, must)); strict_fail += [phr] if must else [] + n = len(qs) + print(f'routing: {len(rows)} rows, {n} phrasings: top-1 {top1}/{n} top-3 {top3}/{n} ({time.time() - t0:.1f}s)') + for phr, exp, order, must in misses: + print(f' MISS{" (must_hit)" if must else ""}: "{phr}"\n want {exp}; got {", ".join(order)}') + return 1 if (a.strict and strict_fail) else 0 + +def main(): + p = argparse.ArgumentParser(description='Advisory authoring audit (maintainer-side; never ships).') + p.add_argument('--root', default=os.path.abspath(os.path.join(HERE, '..', '..'))) + sp = p.add_subparsers(dest='cmd', required=True) + d = sp.add_parser('dupes'); d.add_argument('--top', type=int, default=40) + r = sp.add_parser('routing'); r.add_argument('--strict', action='store_true') + r.add_argument('--phrasings', default=os.path.join(HERE, 'routing-phrasings.tsv')) + a = p.parse_args() + sys.exit(cmd_dupes(a) if a.cmd == 'dupes' else cmd_routing(a)) + +if __name__ == '__main__': + main() diff --git a/tests/advisory/dupes-allow.tsv b/tests/advisory/dupes-allow.tsv new file mode 100644 index 00000000..a4725964 --- /dev/null +++ b/tests/advisory/dupes-allow.tsv @@ -0,0 +1,14 @@ +# dupes-allow.tsv -- pairs `authoring-audit.py dupes` must not report. +# TAB-separated: file_a glob, file_b glob (either order), match, reason. +# match: case-insensitive substring of either chunk, or * for the whole file pair. +# Seeded from the P (pointer/tombstone/boilerplate) labels of the 69 hand-judged pairs, master 484e7ff. +agents/architect-agent.md skills/workflow-structure-rules.md * declared mirror: the stub carries the §12 checklist verbatim +skills/module-completion-loop.md skills/ui-review-loop.md * tombstones: both forward to module-review.md +* * generated from render-routing.sh marker comment +skills/* skills/* **requires:** bash and python 3 standard skill header boilerplate +skills/field-run.md skills/improvement-register.md do not group into a count pointer: field-run cites improvement-register +skills/modularize-domain.md skills/module-folder-convention.md co-location is the point pointer: cites modularize-domain +skills/degrade-to-judgement.md skills/finding-disposition.md harness updates itself both cite skills-over-scripts +skills/degrade-to-judgement.md CLAUDE.md governing rule on where judgement lives both cite skills-over-scripts +skills/brd-to-build-plan.md agents/architect-agent.md n constructs in the diagram, n rows in the plan both cite workflow-structure-rules §12 row 10 +skills/journey-examples.md skills/journey-proof.md after actions and landing guard, before text/trace/data both cite testing-shape §4a rule 7 as canonical diff --git a/tests/advisory/routing-phrasings.tsv b/tests/advisory/routing-phrasings.tsv new file mode 100644 index 00000000..eb7db2ea --- /dev/null +++ b/tests/advisory/routing-phrasings.tsv @@ -0,0 +1,59 @@ +# routing-phrasings.tsv -- labelled phrasings for `authoring-audit.py routing` (CLAUDE.md rule 8). +# TAB-separated: phrasing, expected path, acceptable alternates (comma list or -), must_hit (must_hit|-). +# A must_hit row makes `routing --strict` exit 1 when it misses the top 3. None are must_hit yet. +# Paths are column 2 of bin/lib/skill-routing.tsv. Phrasings are how users and agents actually ask. +page shows 0 to 0 of 0 skills/empty-widget-triage.md skills/fixture-seeding.md - +mx check CE0066 on my access rule bin/bug-lookup.sh bug-logs/mxcli-bugs.md - +should this be a script or a skill skills/skills-over-scripts.md - - +team server push blocked skills/teamserver-alignment.md skills/platform-link.md - +I changed the button colour in the scss and nothing happened skills/learned-css-that-never-applied.md - - +focus outline shows up on every page load, how do I get rid of it skills/learned-css-that-never-applied.md - - +we ported the design system but the app still looks default grey skills/learned-css-that-never-applied.md project-bin/check-design-reaches-app.sh - +I'm about to write a bash checker that flags pages without an H1 skills/skills-over-scripts.md - - +can I just add another regex rule to gate-check for this skills/skills-over-scripts.md - - +is the Orders module done? give it a once-over before we close it skills/module-review.md project-bin/verify-module.sh - +do a final look at every page in the module, not just the ones the tests hit skills/module-review.md - - +just built the customer overview page, grab a screenshot and see if it looks right skills/ui-loop.md - - +the page looks off after my last script skills/ui-loop.md - - +git push to teamserver rejected, which remote is the real one skills/teamserver-alignment.md - - +colleague wants to open it in studio pro from team server, how do we keep both in sync skills/teamserver-alignment.md project-bin/ts-sync.sh,skills/platform-link.md - +ready to push the model up to team server for the first time skills/teamserver-alignment.md skills/platform-link.md - +review the module, did we actually match the wireframes skills/module-review.md project-bin/page-fidelity.js - +Studio Pro won't open the mpr anymore, says it's corrupt skills/mpr-corruption-and-sp-load-errors.md - - +write me a microflow that loops over orders and commits each one skills/microflow-preflight.md skills/learned-microflow-patterns.md - +can I hand the helper one file instead of a reading list project-bin/context-pack.sh - - +grid shows blank cells even though the database has data skills/empty-widget-triage.md - - +playwright says the button never appeared but I can see it on screen skills/e2e-locators-that-lie.md - - +need demo data so the pages aren't empty for the test run skills/fixture-seeding.md - - +where are we and what's next bin/status.sh - - +the user just said the form also needs a phone field skills/change-intake.md - - +lint came back clean with zero findings, feels suspicious skills/lint-that-actually-runs.md - - +add a file attachment upload to the claim form skills/learned-file-upload-widget.md - - +datagrid 2 or the old datagrid for the overview page skills/learned-dg2-patterns.md - - +popup opens a second copy of itself when validation fails skills/learned-popup-navigation.md - - +works on my laptop, the customer tests on the cloud node tomorrow skills/deploy-to-sandbox.md - - +where should the API key for the payment provider live skills/learned-constants-and-secrets.md project-bin/constants-audit.sh - +exec.sh refused to run my script, says something about a lock skills/iterative-build-loop.md bin/exec-approval.sh - +permission denied on my npx command even though it's allowlisted skills/agent-permission-friction.md - - +should this be one module or split into two skills/modularize-domain.md skills/module-dependency-review.md - +we only have a Word spec and some Excel sheets, no code skills/kb-generation.md skills/document-discovery.md - +need an approval user task with an escalation timer skills/learned-workflow-patterns.md skills/workflow-structure-rules.md - +call the vendor's REST API for the first time skills/rest-integration-first-time-right.md - - +app runs in a container, stakeholder needs a public link skills/preview-over-hub-tunnel.md bin/run-hub.sh - +doctor.sh shows red on mxbuild skills/doctor-triage.md - - +the test wrote a row but my DB query shows nothing skills/learned-local-db-confusion.md - - +the tests are green but would they even catch a break skills/journey-proof.md skills/testing-shape.md,skills/learned-detection-gaps.md - +add a chat assistant panel to the dashboard skills/mendix-agent-ui.md - - +should I file this against mxcli or open a toolkit PR skills/upstream-feedback.md - - +sidebar collapses but the menu labels get cut off skills/learned-sidebar-collapse-icons.md - - +how much space between sections on a page skills/design-spacing.md - - +client app, we want to show it at a conference without their logo or data skills/anonymize-client-app-for-demo.md - - +the subagent says all tests passed skills/tool-output-is-not-ground-truth.md - - +about to go live and move the old data over, what's left to decide skills/checkpoints/checkpoint-cutover.md - - +save button on the popup commits but the user never sees anything happen skills/learned-popup-feedback-pattern.md - - +the learned rule says use MCP for this but maybe the new mxcli fixed it skills/retesting-learned-rules.md - - +the existing app has no docs and I need to add a feature to it skills/existing-app-change.md - - +send the guide and screenshots to the customer skills/share-demo-package.md - - +half the buttons on the page do nothing when clicked skills/wiring-sweep.md - - +how do I hand the built model to someone who uses Studio Pro skills/handoff-to-studio-pro.md - - +the order count on the dashboard comes from the wrong place, the page shows the first record not mine skills/interaction-model.md - - diff --git a/tests/wave2/run-all.sh b/tests/wave2/run-all.sh index ec28af87..ea1bcb9d 100755 --- a/tests/wave2/run-all.sh +++ b/tests/wave2/run-all.sh @@ -44,7 +44,7 @@ for t in "$DIR"/test-*.sh; do # Both spellings appear in the fixtures: `/path/to/gate-check.sh` and `[path-to-init-project.sh]`. # The second one matches the filename pattern whole, so the prefix has to come off explicitly. subj_name=$(grep -iEm1 '^# *usage:' "$t" \ - | grep -oE '[A-Za-z0-9_.-]+\.(sh|md)' | grep -v "^${name}$" | head -1 \ + | grep -oE '[A-Za-z0-9_.-]+\.(sh|md|py)' | grep -v "^${name}$" | head -1 \ | sed 's|^path-to-||') if [ -z "$subj_name" ]; then # Fall back to the "usage:" line the fixture prints when run with no argument. @@ -56,7 +56,7 @@ for t in "$DIR"/test-*.sh; do if [ -n "$subj_name" ]; then for cand in "$TOOLKIT/bin/$subj_name" "$TOOLKIT/bin/lib/$subj_name" "$TOOLKIT/project-bin/$subj_name" \ "$TOOLKIT/skills/$subj_name" "$TOOLKIT/claude-hooks/hooks/$subj_name" \ - "$TOOLKIT/tests/behaviour/$subj_name" "$TOOLKIT/$subj_name"; do + "$TOOLKIT/tests/behaviour/$subj_name" "$TOOLKIT/tests/advisory/$subj_name" "$TOOLKIT/$subj_name"; do [ -f "$cand" ] && { subject="$cand"; break; } done if [ -z "$subject" ]; then diff --git a/tests/wave2/test-authoring-audit.sh b/tests/wave2/test-authoring-audit.sh new file mode 100755 index 00000000..cf432a59 --- /dev/null +++ b/tests/wave2/test-authoring-audit.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# test-authoring-audit.sh — golden check for the advisory routing audit (CLAUDE.md rule 8). +# +# Runs `routing` on the committed phrasings and the real bin/lib/skill-routing.tsv. Calibrated on +# master 484e7ff at top-3 46/55; the floor is 44 so wording churn in one or two trigger rows does +# not go red, while a scorer regression (tokeniser, BM25 maths, column read) does. Also proves the +# exit contract: 0 without --strict even with misses, 1 under --strict when a must_hit row misses. +# +# Usage: bash tests/wave2/test-authoring-audit.sh [path-to-authoring-audit.py] + +set -u +TOOLKIT="$(cd "$(dirname "$0")/../.." && pwd)" +SUT="${1:-$TOOLKIT/tests/advisory/authoring-audit.py}" +[ -r "$SUT" ] || { echo "test-authoring-audit: no script at $SUT"; exit 2; } +. "$TOOLKIT/bin/lib/portable.sh"; require_py +FLOOR=44 +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); printf ' ok %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n %s\n' "$1" "${2:-}"; } +TMP="$(mktemp -d "${TMPDIR:-/tmp}/authoring-audit.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT + +out="$("$PY" "$SUT" --root "$TOOLKIT" routing 2>&1)"; rc=$? +[ "$rc" -eq 0 ] && ok "routing exits 0 (advisory)" || bad "routing exits 0" "rc=$rc" +hit=$(printf '%s\n' "$out" | sed -nE 's/.*top-3 ([0-9]+)\/([0-9]+).*/\1/p' | head -1) +tot=$(printf '%s\n' "$out" | sed -nE 's/.*top-3 ([0-9]+)\/([0-9]+).*/\2/p' | head -1) +if [ -n "$hit" ] && [ "$tot" -ge 55 ] && [ "$hit" -ge "$FLOOR" ]; then ok "top-3 $hit/$tot >= $FLOOR" +else bad "top-3 >= $FLOOR of >= 55 phrasings" "$(printf '%s\n' "$out" | head -1)"; fi +# Miss listing on a synthetic file (one sure hit, one sure miss), so it keeps testing after the +# real triggers improve and the committed phrasings stop missing. +printf 'doctor.sh shows red on mxbuild\tskills/doctor-triage.md\t-\t-\nzz synthetic miss probe\tskills/no-such-skill.md\t-\t-\n' > "$TMP/m.tsv" +mout="$("$PY" "$SUT" --root "$TOOLKIT" routing --phrasings "$TMP/m.tsv" 2>&1)" +if printf '%s\n' "$mout" | grep -q 'top-3 1/2' && [ "$(printf '%s\n' "$mout" | grep -c '^ MISS')" -eq 1 ] \ + && printf '%s\n' "$mout" | grep -q 'MISS: "zz synthetic miss probe"'; then ok "the one miss is listed, the hit is not" +else bad "the one miss is listed, the hit is not" "$(printf '%s\n' "$mout" | head -3)"; fi + +printf 'should this be a script or a skill\tskills/no-such-skill.md\t-\tmust_hit\n' > "$TMP/p.tsv" +"$PY" "$SUT" --root "$TOOLKIT" routing --phrasings "$TMP/p.tsv" >/dev/null 2>&1; rc=$? +[ "$rc" -eq 0 ] && ok "must_hit miss without --strict exits 0" || bad "must_hit miss without --strict exits 0" "rc=$rc" +"$PY" "$SUT" --root "$TOOLKIT" routing --phrasings "$TMP/p.tsv" --strict >/dev/null 2>&1; rc=$? +[ "$rc" -eq 1 ] && ok "must_hit miss under --strict exits 1" || bad "must_hit miss under --strict exits 1" "rc=$rc" + +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" -eq 0 ]