Extraction report reads extraction-quality*.json (#151); token-burn cost/windows/effort + last-record usage fix (#296)

[MendixMau]

#151: the extraction report ignored extraction-quality*.json

Problem. extractor-quality-loop.md treats the validator's extraction-quality.json as the proof that an extraction reached the 95% gate. bin/extraction-report.sh, the Stage 1 surface, never read that file. In one field build a 100% score on 7 dimensions showed up as "2 pass, 18 skipped", with no quality section at all.

Fix (bin/extraction-report.sh).

• The report now looks for extraction-quality*.json in four places: analysis/<src>/ (the location the skill names), analysis/<src>/extracted/ (where the field project wrote it), and the same two inside the KB.
• Each file it finds gets its own section. The dimension table has one row per key in the file's dimensions, showing score, found / expected, gaps and a flag. Dimension keys come from the file, so a non-code source shows its own dimensions.
• Above the table: overall score against threshold, passed, selfGraded and confirmedZeros. The same data is in --json (extractionQuality) and there is one summary line in the prose output.
• Verdicts only apply rules that the skill states and that can be seen in the file itself:
  • fault: selfGraded together with passed:true ("self-graded pass"); passed:true with a score under its own threshold; a 0/0 dimension that is not in confirmedZeros (the Suspicious-Zero Gate); unparseable JSON.
  • manual: the run is not passed, or it is self-graded and not passed.
  • pass: otherwise.
• A score file inside <kb>/extracted/ is no longer read as an item array. On master it shows up as "would not parse" and weakens the zero corroboration.
• skills/extractor-quality-loop.md: one short "Where it is read" pointer. This is an on-demand skill, so the baseline word budget is untouched.

Missing score file (review fix): a code-extracted or mixed KB with no score file gets a manual section, "not produced yet — no shipped pipeline writes a validator; build one per extractor-quality-loop.md, or waive the gate". It is not a fault, because nothing mandatory produces the file (CLAUDE.md rule 5). A score file that is present but wrong is still a fault. Documents KBs get skipped. The gate and the exit code are unchanged.

Golden input: there is no captured extraction-quality*.json anywhere in this repo, because no pipeline ships a validator. The fixture reads the example object from the skill's "Output Format" section at run time, so it is not copied in. The variants are mutations of that object. A field run against a real validator output is still owed.

#296: token-burn cost, windows, agent time, human prompts

Fix (bin/token-burn.sh), all opt-in.

• --rates FILE gives cost per model and usage class. Prices are per Mtok, the file holds model globs (first match wins, * as catch-all), and the currency label defaults to EUR. No price is built in. A model with no matching line is printed UNPRICED and the total is marked incomplete; it is never counted as 0.
• --windows FILE takes <start> <name> lines (each window runs until the next one starts) and prints per window: messages, headline tokens, cache-read, cost, agent time and prompts.
• --effort / --gap MIN (default 5):
  • Agent working time is the sum of main-thread gaps of at most N minutes, leaving out gaps that end at a human prompt.
  • Human prompts are counted by origin.kind == "human", falling back to non-< text for older records. Tool results, isMeta, compact summaries, task notifications, peer messages and sidechain records are excluded, and each uuid counts once.
• Output is aggregates only; no prompt text is ever printed. --brief is unchanged.

Unchanged without the flags. Before the last-record fix below was added, screen, --json and --brief output matched master byte for byte, both on the captured fixture and on a live transcript. Fixture T9 asserts that no new section or key appears.

Field run on a real multi-day Claude Code transcript (harness 2.1.29x): the script counted 89 human prompts, and an independent count of origin.kind=="human", non-meta records also gave 89. It also reported 5h25m agent time, plus a windows table and a cost table at made-up rates.

Fixture honesty. The captured fixture has no main-thread human prompts. I tried to capture and scrub a real session transcript, but that commit was refused. So tests/wave2/fixtures/token-burn/effort/s-3.jsonl is hand-built from a profile of real record shapes; only keys and value kinds were read. CAPTURE.md says this plainly. Agent time on the captured s-1 main thread is real, and every expected number was computed independently.

Lead-requested fix: usage per message.id = per-counter max over its records

The header claimed that every streamed record of a message carries the same usage. That is false for subagent transcripts: early records carry a partial output_tokens. The script now keeps the first record's time and the last record's usage.

Before → after:

• Captured fixture, subagent model: output 60 → 1,559; headline 59,858 → 61,357; stage 2 69k → 70k.
• Live multi-agent session: subagent output 37,791 → 84,915.
• Main thread: identical first to last, so main-thread numbers do not change.

Up to 7× undercount was measured elsewhere on 2.1.296. Token counts on subagent-heavy projects will rise after sync. Fixture expectations and CAPTURE.md were re-derived. Found while promoting #171 (measured-claims §7).

Checks

• tests/wave2/test-token-burn.sh bin/token-burn.sh: 63/63 pass. On master: 29 fail. The new zero-usage case fails on the first commit of this PR.
• tests/wave2/test-extraction-quality-report.sh bin/extraction-report.sh (new): 27/27 pass. On master: 22 fail.
• check-scripts, check-portability, check-no-private-citations, leak guard (LEAKGUARD_ALLOW_NO_DENYLIST=1) and check-pr-discipline origin/master: all clean. The forbidden-word grep over added lines returns 0 hits.
• Routing was not touched, so render-routing --check was not needed.
• Full suites were not run.

Field-run status

• token-burn: field run done (above).
• extraction-report: owed. It needs a real KB with a real validator output; I did not touch the field project.

Review round (head 1a73996):

• A missing score file is now manual instead of a fault.
• token-burn takes each usage counter's max across a message's records, so a later record with empty usage cannot erase real numbers. The fixture case msg_e1 has a second, zero-usage record.
• The reused order variable is renamed (win_names, msg_order).
• The skill now has a comment noting that a fixture parses its example JSON.

Closes #151
Closes #296

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw

---

Generated by Claude Code

---

Generated by Claude Code

[MendixMau]

Landed on master as part of the integration squash #329 (commit 31e90ae), after independent review — its Review: line is in that commit message. Closing as superseded.

---

Generated by Claude Code